8 Commits

Author	SHA1	Message	Date
vh	5415fd4b30	docs(gemma4-charrp): abliteration measured in isolation — close to free, but it MOVES capability ... Second bench window, operator-authorised after an initial decline and reversal. Stock BF16 against the llmfan46 abliterated BF16: same precision, same pinned upstream template, same 192 items, CoT off. Abliteration was the only axis that moved, which is what the previous run could not claim. Net core cost is 0.6 points — but the headline understates what happened. Capability MOVED rather than degraded: five items lost on contradiction detection, four gained on spatial composition, nearly cancelling. A gain was not predicted by anyone, least of all on that axis. The decision this was authorised to settle: llmfan46 stands as the trainee base. No case for re-staging on TrevorJS at KL 0.09 over 0.6 points — the KL gap between the builds is smaller than the gap this measurement failed to find. Both limits recorded rather than buried, per brokkr-smithy-dev: the swings are ~5 and ~4 items at n=32, so the -15.6/+12.5 percentages read more precisely than the measurement supports and only marginals were run; and this says nothing about quantization, because the stock-NVFP4 T2 figure came from n=16 against n=32 here — different item counts mean different item sets, so that comparison is n-confounded and is not being made. Turnaround was five minutes rather than fifteen because the gemma4-trainee-bench stack already existed — itself the residue of debugging a 35-restart crash-loop caused by the production compose hardcoding --quantization compressed-tensors. The fix outlasted the incident. gen restored and verified through the gateway; char-rp remains down deliberately; bench stack env reset to the heretic base for the post-tune gate.	2026-08-24 15:39:43 -07:00
vh	14ff4a3f57	docs(gemma4-charrp): stage two abliterated trainee bases; record the endemic stale-template trap ... The operator directed that the ERP/RP trainee base be a low-damage abliterated instruct build rather than the stock checkpoint. Two are now staged under /tank/aimodels/, both BF16, both unquantized, both matching upstream's 51.61 GB / 25.8B shape with only transformers_version differing in config: gemma4-26b-a4b-it-heretic-bf16 llmfan46, Heretic v1.2.0 ARA, KL 0.1237, refusals 3/100 gemma4-26b-a4b-it-abliterated-bf16 TrevorJS, KL 0.09, 1/100 effective and 5/686 cross-dataset "Low damage" was treated as a measurable claim rather than a description: the field spreads from KL 0.09 to 0.4118 and the table is in the README so the next choice is made on numbers. Fleet anchor for reading them — our own abliteration work found Heretic at KL 0.12 preserved the MTP head at 83.7% acceptance, so both staged builds sit inside an already-validated band rather than past it. huihui-ai is rejected despite its reputation: no published metrics, its own card calls the method a crude proof-of-concept, it abliterates both thinking and non-thinking modes, and its parameter count runs ~738M over upstream. The operator's independent read matched. The more durable finding is the chat template. NOT ONE third-party Gemma-4 derivative pulled here ships upstream's — three independent repos carry the identical stale 266-line file (sha 58c66fdee4afa297), llmfan46 carries a third 365-line variant, and only the RedHatAI NVFP4 build matches upstream's 6a1015c47ccfcfa6. It propagated through the ecosystem rather than one packager slipping, and it is now recorded as a class rather than as the single incident that surfaced it during the A16 control staging. That matters twice over and silently both times: serving a mismatched template renders a different prompt, which is why production pins it; and training through `base/chat_template.jinja` means training on a different prompt format than production serves — train/serve skew with no error, presenting as a tuning failure. brokkr-smithy-dev has been warned on the training side while the harness contract is still early enough to amend.	2026-08-24 13:52:00 -07:00
vh	8d6a9390de	docs(gemma4-charrp): RETRACT the contradiction-deficit claim — the item was ill-posed ... Supersedes what commit 3446367 recorded. That message stated the A16 control "settles a question" and quoted 12% contradiction detection against gen's 81%. The quantization half of it stands; the deficit it was measuring does not exist. brokkr-smithy-dev retracted the finding after the operator asked to see the individual items. The task presented two mutually contradicting statements and asked for "the contradicting statement" — but CONTRADICTION IS SYMMETRIC. Neither statement was more the contradicting one, the model had no way to know which had been inserted, and it consistently named the absolute claim: a defensible reading that the labelling scored wrong on every single item. The tell was there and both of us walked past it: 12% on a five-option task is BELOW THE 20% CHANCE FLOOR. A below-chance score indicts the instrument before it indicts the model. That should be the first reaction to a below-chance result, not a late one, and it is now written into the README as such. Retracted: "the model owns the contradiction deficit"; "domain tuning costs 43 points of contradiction detection" (which on a sound instrument does not shrink but REVERSES); and every pre-fix T2 number for Gemma-4, MeroMero-v2, sec and gen. A second defect surfaced during the fix — all generators shared one RNG, so rewriting one task reshuffled every task after it. What survives is real and worth separating out: the A16 control result holds. Activation precision is close to free on this workload, every other task identical across the W4A4 and W4A16 builds. The two staging confounds caught before the run — the two Hub repos named NVFP4A16 that declare 4-bit activations, and the stale chat template — were independent of the item defect and remain load-bearing. On the corrected instrument Gemma leads the very axis it was suspected of failing (94% against sec's 81% and gen's 50%); its actual weak axis with thinking off is spatial composition at 69%. Recorded as a dated superseded-claims table rather than a silent edit, per the repo's quant-work convention, so notes elsewhere stop misleading people.	2026-08-24 13:07:58 -07:00
vh	3446367d5e	feat(gemma4-charrp): pin the chat template; A16 control run executed and reverted ... The template is now passed explicitly, defaulting to the A4 build's chat_template.jinja. That is a no-op for what is served — the A4 build ships that exact file, byte-identical to upstream google/gemma-4-26B-A4B-it once trailing newlines are normalised — and it permanently closes the class of bug found while staging the control: the A16 build ships a stale 266-line template against upstream's 390, with the thinking path built differently and no `thinking` property in its tokenizer_config response_schema. Serving each build with its own template would have moved a second axis. The control ran on the operator's greenlight and has been reverted. Seat is back on the W4A4 build, healthy, RestartCount 0, both aliases verified through the gateway — char-rp returns content with reasoning_content empty, char-rp-reasoning returns both. Result, since it settles a question this repo's config now encodes: activation precision does NOT explain the contradiction-detection deficit. Contradiction detection moved 12% -> 19% between W4A4 and W4A16, which at n=16 is 2/16 -> 3/16 — one item — against gen's 81% on identical items. Every other task is identical across the two builds and the core difference is 2.6 points carried almost entirely by two single items. brokkr-smithy-dev pre-registered that a null result would be the robust branch, because a hidden third axis would tend to create a delta rather than suppress one, so the conclusion survives the residual doubt neither side could close without a dequantization pass. The practical upshot for future scheme choices: W4A4 costs less on this workload than the caution warranted. The caution was still correct to have. Displaced production for 3.7 seconds of measurement plus two container recreates. The A16 build and the BF16 tuning base both stay on disk with the runbook in the stack README, so re-running is a two-minute flip.	2026-08-24 13:03:19 -07:00
vh	1bd90eaacc	docs(gemma4-charrp): stack README — the three model dirs, the A16 control runbook ... The stack had no README and now carries three model directories that look interchangeable and are not: the BF16 QLoRA base that cannot be served here, the W4A4 quant that is served, and the W4A16 build that exists solely as an activation-axis control. Writing down which is which, and why, before someone "simplifies" the compose to the BF16 path and rediscovers the OOM. Also captures the A16 control procedure end to end, including the two confounds found while staging it — the two Hub repos named NVFP4A16 that declare 4-bit activations, and the stale chat template the real one ships — and the fact that overriding the template is safe because the tokenizers are identical. Both sides have now cross-checked this: brokkr independently diffed every non-quantization config field of both builds against the upstream BF16 and found only transformers_version differing. Residual risk recorded rather than hidden: config identity is not weight identity and nobody has done a dequantization pass. The Gemma-4 flags are documented as architecture-level rather than checkpoint-level, since that is why they survived the seat swap unchanged, and the enable_thinking:false pin is called out as mandatory rather than stylistic — without it every plain prose turn lands in reasoning_content with a null content and every consumer breaks. Notes the non-termination defect with thinking on (32 of 96 calls truncating at 12k tokens, all 16 constraint items among them, reasoning sound right up to the point it fails to stop) and why VLLM_USE_V2_MODEL_RUNNER=0 is deliberately not applied to a seat whose production mode is thinking-off. No live change: the seat is still serving the W4A4 build. Displacing it for the control run is an operator decision and is still open.	2026-08-24 12:34:50 -07:00
vh	24e8826219	docs(gemma4-charrp): the A16 control needs a chat-template override, not just a path swap ... Pre-flighting the staged A16 build before handing it to brokkr-smithy-dev's battery found a second axis hiding inside what was supposed to be a single-variable control. The A16 build ships a STALE chat template. Verified by hash against the upstream weights on the same disk: google/gemma-4-26B-A4B-it is 390 lines, the RedHatAI A4 build's is 389 and byte-identical to upstream once trailing newlines are normalised, and the prithivMLmods A16 build's is 266 and is not. The delta is not cosmetic — upstream and A4 open the thinking path with `{%- set enable_thinking = enable_thinking | default(false) -%}` and branch off it, while the A16 template has no such set and guards with `enable_thinking is defined and enable_thinking` instead. tokenizer_config.json corroborates: A4's response_schema carries a `thinking` property, A16's has only role and content. That build was quantized from an older revision of the checkpoint. Served with its own template, the A16 arm would render a different prompt for identical messages, and a contradiction-detection delta could be attributed to activation precision when it was the template. That is the same failure class as the misnamed-A16 repos — a field nobody validated, believed because the name looked right — one layer further down, and it would have produced a result that looked like a finding. Overriding is safe because the tokenizers agree: vocab identical at 262,144 entries, added_tokens identical, so the same template over the same vocab renders the same token ids. Everything else pre-flights clean — both artifacts complete with no missing shards, generation_config.json byte-identical. Seat NOT flipped; displacing production for the bench window is the operator's call and is still open.	2026-08-24 12:32:03 -07:00
vh	f509668e45	docs(gemma4-charrp): record the A16 activation-axis control and how to run it ... brokkr-smithy-dev's first battery on the new seat scored 12% on contradiction detection with CoT off, against gen's 81%, while state tracking, deterministic constraint following, long-context state and the confabulation control all sat at 100%. That is not general degradation — it is the shape 4-bit input activations produce on the most reasoning-dense task, which is exactly the confound flagged when the W4A4 quant was chosen. They have recorded the finding as CONFOUNDED rather than reporting it as a property of Gemma, and asked for an A16 build to separate the two. No quant run was needed: a genuine A16 build of the same checkpoint already exists on the Hub and is now at /tank/aimodels/gemma4-26b-a4b-it-nvfp4a16 (prithivMLmods, compressed-tensors, nvfp4-pack-quantized, input_activations null, 17 GB). Same weights, same loader as the live seat, one axis moved. ⚠ Two other repos would have answered the question wrongly and the note in the env template says so: bg-digitalservices and ManniX-ITA both publish "Gemma-4-26B-A4B-it-NVFP4A16" whose config declares input_activations num_bits 4 — W4A4 under an A16 name, via modelopt, both 16.46 GB against the real one's 17.93. Using either and seeing T2 stay at 12% would have wrongly convicted the model. Running it is a one-line GEMMA4_MODEL swap plus a recreate, because there is no room for a concurrent seat: GPU0 has 3.5 GiB free with gen and the A4 seat on it, and GPU1's 19.4 GiB against ~18 GiB of weights is the same sub-GiB headroom that crash-looped the predecessor this morning. Port and both aliases are unchanged either way, so no consumer config moves.	2026-08-24 12:28:17 -07:00
vh	27155c0f3b	feat(char-rp): swap the seat to the Gemma-4 26B-A4B MoE, NVFP4, same port ... Straight-across replacement of the dense G4-MeroMero-v2-31B-NVFP4A16 seat with google/gemma-4-26B-A4B-it on ana-ml2 GPU0. Port, served-model-names and every gateway route are unchanged, so no consumer sees a difference in addressing: `char-rp` -> hosted_vllm/char-rp and `char-rp-reasoning` -> hosted_vllm/char-rp-thinking, both still :8016. The seat's requirements now include chain-of-thought, which makes throughput more critical rather than less — the user waits through the whole reasoning block before the first visible token, and the MoE measures ~114 tok/s @32K against the dense 31B's ~40.7. Both artifacts are on disk and they are NOT interchangeable. The BF16 weights (/tank/aimodels/gemma4-26b-a4b-it-bf16, 49 GB) are the QLoRA tuning base, since QLoRA does its own quantization. They CANNOT be served here: 48.10 GiB of weights against ~49 GiB of free GPU0 leaves nothing for KV cache, and the engine would die at allocation exactly the way the predecessor did this afternoon. The serving copy is RedHatAI/gemma-4-26B-A4B-it-NVFP4 (16 GB), chosen over the other -it quants because it is compressed-tensors (nvfp4-pack-quantized) — the same loader path the outgoing seat used — from the llm-compressor team at 357k downloads. The nvidia/ repo is the base rather than -it, and the thinking channel lives in the instruction-tuned weights. Smaller weights at the same 0.47 memory budget buy a much larger KV pool: 27.37 GiB and 1,724,110 tokens, against the predecessor's 371,023 at the same budget. That is 6.5 full-length 262K sequences concurrent rather than 1.4. The gemma4 tool-call parser, reasoning parser and the enable_thinking:false default all carry over unchanged — they are architecture-level, not checkpoint-level. The --chat-template override does NOT carry over: MeroMero pointed at a jinja hand-patched against that checkpoint, and this model ships its own. Verified that dropping it did not reintroduce the failure that flag existed to prevent — non-thinking prose lands in content with reasoning_content empty, and the thinking alias populates reasoning_content with content carrying the answer. ⚠ Scheme differs from the incumbent and the bench should say so: this quant declares 4-bit input activations (W4A4) where the outgoing seat was NVFP4A16. Faster, and not like-for-like on the activation axis. meromero-charrp is retained stopped in `created` state and relabelled to AI - Dormant, per the house rollback pattern. Both stacks want :8016, so rolling back means stopping the gemma4 seat first.	2026-08-24 12:02:59 -07:00