vh
725c8fdf9e
Quantized the L35 abliterated model to mixed NVFP4 and measured MTP acceptance
end to end. The experiment's whole premise: Heretic (the incumbent gen seat)
leaves the MTP head a byte-identical base graft its wrapper never loads, whereas
Robinson abliterates the MTP head in-band — the question was whether that in-band
edit survives well enough to spec-decode. It does, better than the graft:
MTP acceptance 59.1% median (51-65%, 8 cache-busted topics) vs incumbent ~47%
decode 118.7 tok/s median (faster; image-confounded, read as not-worse)
abliteration survives quant (creative refusals drop, self-harm guardrail
intact, coherent)
Output at /tank/aimodels/qwen38-27b-coldfusion-L35-nvfp4-mixed (22.5 GB). Result
JSON in bench/. NOT cut over — the incumbent seat is untouched; making L35 the gen
seat is a separate decision needing the full Stage-3 gate + real multi-turn hold.
Two env foot-guns hardened along the way:
- quant_mixed_nvfp4.py now promotes text_config attention fields
(num_attention_heads etc.) to the top-level config for the oneshot, then
restores. transformers 5.10 / llmcompressor 0.12 (this venv moved under us
since the Aug-15 heresy quant) no longer delegate the top-level lookup, so
oneshot raised "Cannot determine num_attention_heads". Same "the fight is the
environment" pattern as the abliteration capture.
- a sub-~23GB quant saves as a single model.safetensors with no index, so the
post_quant MTP graft needed an index built first — from the safetensors header,
not safe_open (which mmaps the whole shard and ENOMEMs on ZFS).
post_quant grafted the abliterated MTP (15 tensors, 849 MB) and re-injected
re:^mtp.* into quantization_config.ignore (llm-compressor pruned it again — the
two-rounds-lost 0%-MTP bug, fired and repaired as designed). Probe served on the
pinned nightly (#51113 qwen3_5_mtp fix) to match the live seat's vLLM.
29 lines
651 B
JSON
29 lines
651 B
JSON
{
|
|
"tag": "coldfusion-L35",
|
|
"model": "probe",
|
|
"tok_s_median": 118.71736157977526,
|
|
"tok_s_mean": 118.55983517633987,
|
|
"tok_s_min": 109.41762922436195,
|
|
"tok_s_max": 127.07885746551419,
|
|
"mtp_accept_median": 0.5905651340996169,
|
|
"rates": [
|
|
118.19663140596533,
|
|
115.20398398977953,
|
|
113.01193271508572,
|
|
109.41762922436195,
|
|
127.07885746551419,
|
|
119.25802285580335,
|
|
119.23809175358517,
|
|
127.07353200062373
|
|
],
|
|
"accs": [
|
|
0.5839080459770115,
|
|
0.5592841163310962,
|
|
0.5482456140350878,
|
|
0.5138004246284501,
|
|
0.654320987654321,
|
|
0.5972222222222222,
|
|
0.5972222222222222,
|
|
0.654320987654321
|
|
]
|
|
} |