feat(coldfusion-abliteration): Heretic-300 — 8/100 refusals at KL 0.0136, beats the heresy bar 3.6x

Ran Heretic v1.4.0's 300-trial TPE search on Cold-Fusion. Best trial scores 8/100 refusals at KL 0.0136 against a 98/100 base, versus absolute-heresy at 29/100 and our hand-tuned Robinson L35 at 72/100 / KL 0.0116 — i.e. 64 fewer refusals for the same damage. Hand-verified coherent: correct arithmetic with shown working, clean code, 66-167 word prose across nine probes. Durable findings: - direction_scope=0 (single shared direction) is decisive on this merged base: n=129, best 8/100. Per-layer directions n=131 never beat 52/100 despite a better median. Points against the multi-direction intuition for a diffuse direction (our two-template |cos| is 0.62 vs Robinson's 0.99 on stock). - Aggression is not the lever. r(KL, refusals) = -0.561 over 261 trials; the KL<0.02 band contains both the worst results (median 87/100) and the single best. A KL 0.3554 trial scored worse than one at 0.0193. - PR #317 confirmed: Heretic silently drops the MTP head on save. Source 1199 tensors -> export 1184, all 15 mtp.* gone, vision 333/333 intact, exit 0, no warning. This is also why absolute-heresy ships a byte-identical MTP head — a bug, not a design choice. Always diff tensor keys after a Heretic export. - Heretic's recovered direction carries 6.18% of its energy in sink dim 3994, versus 0.094% for our L35 and 1.97% for the L39 we rejected as brick-inducing. It survives that only because of magnitude-preserving ablation (row_normalization=FULL); our plain projection has no such protection, so the sink screen correctly refused the in-band MTP graft. Same direction, different operation. MPOA is the prerequisite for in-band MTP on a Heretic trunk. - Heretic's edit is recoverable from weights: delta is rank-1 (s2/s1 ~ 0.010), SVD gives the direction, norms give per-layer weights (1.08 -> 1.34, i.e. over-projection). Cross-layer |cos| agreement 0.9903 independently confirms the single-direction result. New tooling in services/coldfusion-abliteration/: kl_divergence.py first-token KL, class-split, zero noise floor catatonia_gate.py 12 probes x 220 tokens, prints every completion heretic_export.py PTY driver; selects by measured value, never by menu position — Heretic's resume prompt puts "delete the checkpoint and all results" one arrow-key from the target graft_mtp.py recovers the trunk direction by SVD; --pristine for the safe path when the sink screen refuses Also adds quant playbook 3.13: the NVFP4 recipe sets observer="imatrix_mse" but llm-compressor has always silently fallen back to uniform MSE for want of importance data — on this build and on the incumbent. Existing A/B comparisons stay valid since every build shares the fallback. Parked as id 42. Guardrail note: this build has lost the self-harm guardrail that the Robinson L35 build retained. Restoration is the operator's own work item.
2026-08-20 22:51:56 -07:00
parent 78484ac87d
commit f90a5025de
6 changed files with 927 additions and 0 deletions
@@ -349,6 +349,49 @@ Two gate corollaries learned in the same session:
  the 8-byte length prefix + JSON header, never `safe_open`, which mmaps the whole shard
  and ENOMEMs on ZFS (§ *Avoid mmap on `/tank`*).

+### 3.13 ⭐⭐ The observer you ASKED for is not necessarily the observer you GOT
+
+`quant_mixed_nvfp4.py` sets `observer="imatrix_mse"` on the NVFP4 W4A4 group. It has
+**never once been used.** llm-compressor looks for importance data, finds none, and
+silently degrades:
+
+```
+_get_validated_importance | WARNING - imatrix_mse: no importance data available.
+                                      Falling back to uniform MSE.
+```
+
+Confirmed on the 2026-08-20 09:59 incumbent quant **and** the 22:45 Heretic-300
+quant; `find /tank/aimodels -iname "*imatrix*" -o -iname "*importance*"` returns
+nothing. Every NVFP4 build in the fleet has run uniform MSE while the recipe claimed
+importance weighting.
+
+**Why it went unseen for months:** the warning scrolls past inside a tqdm progress
+bar during a ~20 minute quant. It is only visible if you read the log while it runs.
+
+**The generalisable rule, which is bigger than imatrix.** A quantizer, optimiser or
+observer that *silently falls back to a weaker default* is a whole class of invisible
+quality loss — the config is accepted, nothing errors, the artifact benchmarks
+plausibly, and you never learn you got the cheap path. So:
+
+- **Grep the quant log for `WARNING`, `Falling back`, `not available`, `ignoring`
+  before trusting an artifact.** Make it a step, not a habit.
+- **Assert the effective setting, never the requested one** — the same rule as
+  [[feedback_assert_effective_value_not_substring]], applied to quantizer internals
+  rather than config files.
+- If the fallback turns out to be unavoidable in your toolchain version, **change the
+  recipe to say what it actually does.** A recipe line that silently lies is worse
+  than one that admits a limitation.
+
+⚠️ **Do not "fix" this by assuming an imatrix would help.** Verify first that your
+llm-compressor version can consume an externally supplied importance matrix at all,
+and in what format. Parked as `park/nvfp4-recipe-asks-for-imatrix-mse-but-silently-2`
+(id 42) with the calibration corpus that would feed it.
+
+✅ **Comparisons already made remain valid.** Because *every* build shares the
+fallback, the incumbent-vs-candidate A/Bs (47.2% acceptance, PPL 6.910, and the
+2026-08-20 Heretic-300 build) are apples-to-apples. This is unrealised upside, not a
+correction to past numbers.
+
 ---

 ## 4. Pipeline shape