diff --git a/stacks/gemma4-charrp/.env.example b/stacks/gemma4-charrp/.env.example index d03e982..3de76fe 100644 --- a/stacks/gemma4-charrp/.env.example +++ b/stacks/gemma4-charrp/.env.example @@ -26,10 +26,14 @@ GEMMA4_MODEL=/tank/aimodels/gemma4-26b-a4b-it-nvfp4 # ACTIVATION-AXIS CONTROL, for one-off benching — swap this line, recreate: # GEMMA4_MODEL=/tank/aimodels/gemma4-26b-a4b-it-nvfp4a16 # Same NVFP4 weights and the same compressed-tensors loader, but 16-bit -# activations. brokkr-smithy-dev's 2026-08-24 battery scored 12% on -# contradiction detection with CoT off against gen's 81%, while every other -# task sat at 100% — the shape 4-bit input activations would produce on the -# most reasoning-dense task, so the finding is CONFOUNDED until this is run. +# activations. +# +# ⚠ THE ORIGINAL MOTIVATION FOR THIS CONTROL WAS RETRACTED — see the README's +# "Superseded claims" section. The A16 run itself HELD and is the reason this +# path is documented: activation precision is close to free on this battery, +# every other task identical across builds. But the "12% contradiction +# detection" number that prompted it was an artifact of an ill-posed benchmark +# item, not a model property. Keep the path; do not repeat the reasoning. # Two minutes down each way; port and both aliases are unchanged, so no # consumer config moves. # diff --git a/stacks/gemma4-charrp/README.md b/stacks/gemma4-charrp/README.md index 5093be5..445492c 100644 --- a/stacks/gemma4-charrp/README.md +++ b/stacks/gemma4-charrp/README.md @@ -42,11 +42,16 @@ control below is a swap rather than a parallel deployment. ## Running the A16 activation-axis control -Why: brokkr-smithy-dev's 96-item battery scored **12% on contradiction detection -with CoT off**, against gen's 81%, while state tracking, constraint following, -long-context state and the confabulation control all sat at 100%. Not general -degradation — the shape 4-bit input activations produce on the most -reasoning-dense task. Recorded as CONFOUNDED until this run separates the two. +Why it was run, and what it actually settled: a 2026-08-24 battery appeared to +show a large contradiction-detection deficit with CoT off, which had the exact +shape 4-bit input activations would produce on the most reasoning-dense task. +The control ran and found activation precision **close to free** — every other +task identical across the two builds. That quantization result stands. + +⚠ The *deficit* it was chasing does not — the benchmark item was ill-posed. See +"Superseded claims" at the end of this file. The procedure below is kept because +the flip is a genuinely useful capability, not because the original question was +sound. ⚠ **The A16 build ships a STALE CHAT TEMPLATE and the run MUST override it.** Verified by hash 2026-08-24: upstream `google/gemma-4-26B-A4B-it` is 390 lines, @@ -129,3 +134,41 @@ wrong-answer one. `VLLM_USE_V2_MODEL_RUNNER=0` enables `thinking_token_budget` and is the first lever to try if a usable thinking seat is ever wanted — deliberately **not** applied here, since it costs the faster model runner for a mode production does not serve. + +## Superseded claims + +Following the repo convention for quant work: when a recorded claim turns out +wrong, it gets a dated row rather than a silent edit, so older notes elsewhere +stop misleading people. + +| date | claim, as recorded | status | +|---|---|---| +| 2026-08-24 | "Gemma-4-26B-A4B-it scores 12% on contradiction detection with CoT off, against gen's 81% — the model owns the deficit." | **RETRACTED same day.** The benchmark item was ill-posed: it presented two mutually contradicting statements and asked for "the contradicting statement", but **contradiction is symmetric** — neither was more the contradicting one, and the model consistently named the absolute claim, a defensible reading the labelling scored wrong every time. | +| 2026-08-24 | "Domain tuning costs 43 points of contradiction detection." | **RETRACTED.** Rested entirely on the same item. On the corrected instrument the effect does not shrink, it **reverses**. | +| 2026-08-24 | All pre-fix T2 (contradiction) numbers for Gemma-4, MeroMero-v2, `sec` and `gen`. | **VOID.** | + +**The tell, worth internalising: the score was BELOW CHANCE.** 12% on a +five-option task is under the 20% floor. A below-chance score indicts the +instrument before it indicts the model, and that should be the first reaction +rather than a late one. Neither side caught it until the individual items were +read. + +A second defect surfaced while fixing the first: all generators shared one RNG, +so rewriting one task reshuffled every task after it. Each task now seeds from +its own name. + +**What survived, and it is not nothing:** the A16 control result holds — +activation precision is close to free on this workload, with every other task +identical across the W4A4 and W4A16 builds. The two staging confounds caught +before the run (the misnamed A16 repos, the stale chat template) were real and +independent of the item defect. The corrected picture, CoT off, n=96: + +| arm | core | T2 | T6 | median latency | +|---|---|---|---|---| +| `char-rp` (Gemma-4) | 92.5% | **94%** | 69% | 0.24s | +| `sec` | 92.5% | 81% | 81% | 1.11s | +| `gen` | 86.2% | 50% | 81% | 0.33s | + +Gemma leads on the very axis it was suspected of failing. Its actual weak axis +with thinking off is **T6, spatial composition at 69%** — and an earlier run +showed CoT-on takes T6 to 100%, which nothing else benched has managed.