diff --git a/docs/pfi/gen-seat-candidate-eval-2026-08-21.md b/docs/pfi/gen-seat-candidate-eval-2026-08-21.md new file mode 100644 index 0000000..72dc8a7 --- /dev/null +++ b/docs/pfi/gen-seat-candidate-eval-2026-08-21.md @@ -0,0 +1,89 @@ +# Gen-seat candidate evaluation — 2026-08-21 + +Cold-Fusion was abandoned (see `persistent-memory.md`); the seat is on +`qwen38-27b-heresy-nvfp4-mixed`. Two replacement candidates were put up. All facts +below come from the HF registry and from reading the artifacts directly — the +safetensors headers were fetched with HTTP **Range** requests, so the tensor census +cost about a megabyte rather than a 20 GB download. + +## The candidates + +| | `orcarouter/Qwen3.8-27B-Uncensored` | `preetpatel/…-NVFP4` | +|---|---|---| +| what | BF16 source weights | NVFP4 quant **of orcarouter** | +| size | 55.6 GB | 19.7 GB | +| base | `Qwen/Qwen3.8-27B` (**stock Qwen**) | orcarouter | +| **MTP tensors** | **15 ✓** | **0 ✗** | +| visual tensors | 333 ✓ | 333 ✓ | +| scheme | n/a (bf16) | **NVFP4 W4A4** ✗ | +| `re:^mtp.*` in ignore | n/a | **absent** ✗ | +| traction | 3,278 dl / 60 likes | 36 dl / 0 likes | +| gated | yes — **our token already has access** | no | +| chat template | **sha `c3cf9e34` — byte-identical to the live heresy seat** | same | + +## Verdict: preetpatel is disqualified, on two independent hard failures + +**1. Zero MTP tensors.** Read straight from the safetensors header: 2,672 tensors, +**none** matching `mtp.*`. The author's own `recipe.yaml` asks to ignore +`re:.*mtp.*`, but the written `config.json` contains no mtp ignore entry at all — +while `re:.*visual.*` expanded to 110 explicit entries. That asymmetry is the +signature of llm-compressor pruning an ignore pattern that matched nothing, i.e. +the MTP head was never loaded and never quantized. It is the same +`re:^mtp.*`-pruning trap documented in the playbook, seen from the outside. + +Cost: no speculative decoding. Our seat runs MTP at ~59% acceptance and 118 tok/s; +without it, roughly half the decode throughput. + +**2. NVFP4 W4A4 — 4-bit activations.** `input_activations: num_bits 4, type float`. +This is precisely the AEON failure mode we spent a multi-day saga diagnosing and +purging: the activation-fidelity gradient is W4A4 < W4+FP8 < W4+bf16, W4A4 was +responsible for ~15-20% stochastic degeneration, and W4A4 collapses past ~30k +context. **The gen seat serves 262K.** + +Either failure alone would rule it out. It is also one day old with 36 downloads. + +## orcarouter checks out as a quant source + +Stock-Qwen base (not a reasoning-compression finetune — the trait that sank +Cold-Fusion), Arditi-et-al. single-direction abliteration, MTP and vision both +explicitly preserved and verified at 15/333, chat template byte-identical to the +build we are serving right now, and the gate is already accepted on our token. + +## Third option, noted and not recommended + +`orcarouter/Qwen3.8-27B-Uncensored-FP8` — 76,109 downloads, 693 likes, far more +traction than either candidate. **But 30.9 GB against NVFP4's 22 GB**, and GPU0 is +zero-sum with meromero co-resident: +9 GB of weights comes straight out of the KV +pool, taking it from ~14.4 GiB / 403k tokens to roughly 5 GiB / ~150k — which +breaks 262K context at 1.5x concurrency. Viable only if the seat gives up long +context or meromero moves. + +## The imatrix constraint — read before committing to it + +The operator asked for imatrix if we quant ourselves. **This is not a switch.** + +`quant_mixed_nvfp4.py` already sets `observer="imatrix_mse"` on the W4A4 group and +has **never once used it** — llm-compressor logs `no importance data available. +Falling back to uniform MSE` and proceeds. Playbook §3.13 documents this and warns +explicitly: *do not "fix" it by assuming an imatrix would help; verify first that +your llm-compressor version can consume an externally supplied importance matrix at +all, and in what format.* Parked as `park/…imatrix-mse…` (id 42) with the +calibration corpus that would feed it. + +Also note the W4A16 portions of the mixed recipe are **data-free by construction** — +llm-compressor infers `DataFreePipeline` for weight-only quantization and ignores +calibration data entirely. Imatrix can only ever bite on the W4A4 MLP group. + +So "quant with imatrix" is two projects: an unscoped capability investigation, and +then the ~2h quant. Recommendation is to decouple them — ship the proven recipe +first, run imatrix as its own bounded experiment. Every A/B we hold is +uniform-MSE-to-uniform-MSE, so a non-imatrix build stays directly comparable to +heresy's PPL 6.910 / 47.2% acceptance. + +## Mandatory step if we pull + +Run `services/gen-seat-mixed-quant/bench/think-leak/think_prior.py` on the bf16 +**before any GPU time**. It is a ~10s CPU measurement and it is the gate that would +have disqualified Cold-Fusion before its 300-trial study ever ran. Prior is +favourable — stock-Qwen base, template identical to heresy, which measures <0.002 +against Cold-Fusion's 0.185 — but measure, don't assume.