docs(gemma4-charrp): RETRACT the contradiction-deficit claim — the item was ill-posed

Supersedes what commit 3446367 recorded. That message stated the A16 control
"settles a question" and quoted 12% contradiction detection against gen's 81%.
The quantization half of it stands; the deficit it was measuring does not exist.

brokkr-smithy-dev retracted the finding after the operator asked to see the
individual items. The task presented two mutually contradicting statements and
asked for "the contradicting statement" — but CONTRADICTION IS SYMMETRIC.
Neither statement was more the contradicting one, the model had no way to know
which had been inserted, and it consistently named the absolute claim: a
defensible reading that the labelling scored wrong on every single item.

The tell was there and both of us walked past it: 12% on a five-option task is
BELOW THE 20% CHANCE FLOOR. A below-chance score indicts the instrument before
it indicts the model. That should be the first reaction to a below-chance
result, not a late one, and it is now written into the README as such.

Retracted: "the model owns the contradiction deficit"; "domain tuning costs 43
points of contradiction detection" (which on a sound instrument does not shrink
but REVERSES); and every pre-fix T2 number for Gemma-4, MeroMero-v2, sec and
gen. A second defect surfaced during the fix — all generators shared one RNG, so
rewriting one task reshuffled every task after it.

What survives is real and worth separating out: the A16 control result holds.
Activation precision is close to free on this workload, every other task
identical across the W4A4 and W4A16 builds. The two staging confounds caught
before the run — the two Hub repos named NVFP4A16 that declare 4-bit
activations, and the stale chat template — were independent of the item defect
and remain load-bearing. On the corrected instrument Gemma leads the very axis
it was suspected of failing (94% against sec's 81% and gen's 50%); its actual
weak axis with thinking off is spatial composition at 69%.

Recorded as a dated superseded-claims table rather than a silent edit, per the
repo's quant-work convention, so notes elsewhere stop misleading people.
This commit is contained in:
vh
2026-08-24 13:07:58 -07:00
parent 3446367d5e
commit 8d6a9390de
2 changed files with 56 additions and 9 deletions
+8 -4
View File
@@ -26,10 +26,14 @@ GEMMA4_MODEL=/tank/aimodels/gemma4-26b-a4b-it-nvfp4
# ACTIVATION-AXIS CONTROL, for one-off benching — swap this line, recreate:
# GEMMA4_MODEL=/tank/aimodels/gemma4-26b-a4b-it-nvfp4a16
# Same NVFP4 weights and the same compressed-tensors loader, but 16-bit
# activations. brokkr-smithy-dev's 2026-08-24 battery scored 12% on
# contradiction detection with CoT off against gen's 81%, while every other
# task sat at 100% — the shape 4-bit input activations would produce on the
# most reasoning-dense task, so the finding is CONFOUNDED until this is run.
# activations.
#
# ⚠ THE ORIGINAL MOTIVATION FOR THIS CONTROL WAS RETRACTED — see the README's
# "Superseded claims" section. The A16 run itself HELD and is the reason this
# path is documented: activation precision is close to free on this battery,
# every other task identical across builds. But the "12% contradiction
# detection" number that prompted it was an artifact of an ill-posed benchmark
# item, not a model property. Keep the path; do not repeat the reasoning.
# Two minutes down each way; port and both aliases are unchanged, so no
# consumer config moves.
#