920f37c6c22a4b46fa35baebd007bc6e4c41747a
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 vh
|
c8f128bdff |
feat(gen-seat): quant orcarouter — its MTP head is already Robinson-abliterated in-band
Pulled orcarouter/Qwen3.8-27B-Uncensored at rev 9878936b (55.5 GB, gated, our token has access) and built /tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed (23.4 GB, mixed NVFP4+FP8). Verified, not yet cut over. The operator asked whether we could apply the Robinson path to the MTP head. We cannot, because the author already did. compare_mtp_head.py against the verbatim base graft: 13 of 15 tensors byte-identical, exactly 2 differ -- mtp.layers.0.self_attn.o_proj.weight and mtp.layers.0.mlp.down_proj.weight, which are precisely the two residual writers our own abliterate.py targets (EXPECT_MTP_WRITERS = 2). Reverse-engineered the edit from the weights alone (mtp_delta.py, added here): sigma2/sigma1 = 0.0164 on BOTH tensors rank-1, a single-direction projection |cos| between the two recovered dirs = 1.0000 ONE shared direction ||delta||/||W|| = 1.42% and 1.41% a gentle, consistent projection sink energy dim 3994 = 0.0000% sink-clean; Heretic's was 6.18% That is the Robinson in-band MTP abliteration, already applied, with a direction that passes our sink screen outright. Nothing to do but preserve it, and the quant carries it byte-identically. This is the configuration the entire Cold-Fusion experiment was designed to test and never cleanly delivered. The new format screen paid for itself on its first real use: think_prior.py on the bf16 BEFORE any GPU time gave P(<think>) = 1.23e-06 at rank 52, against Cold-Fusion stock 0.1850 and h300 0.2216. Roughly 150,000x cleaner. Two durable findings about the pipeline itself: The quant needs ~17 GB, not a whole card. It ran entirely in GPU1's spare 16 GB with ZERO production seats stopped -- the h300 run's "stop BOTH GPU0 seats" was never necessary, it simply had a free card by coincidence. The first attempt OOM'd by 2.37 GiB at layer 64 of 65 with 3.57 GiB reserved-but-unallocated, which is fragmentation, and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True closed it. post_quant.py now builds a missing output index from the safetensors headers. A sub-23 GB quant saves one bare shard with no index, and post_quant needs one; this has broken three separate rounds and been hand-fixed every time. The header is read by struct-unpacking the u64 length and parsing the JSON -- never safe_open, which mmaps the whole 22 GB shard and ENOMEMs on ZFS. Artifact verified: mixed-precision, 1968 tensors, 15 mtp, 333 visual, re:^mtp.* present in the ignore list (llm-compressor pruned it as always), preproc restored. Imatrix deferred per operator; the log confirms the usual uniform-MSE fallback, so this build stays apples-to-apples with heresy's PPL 6.910. |
||
|
vh
|
993421bf59 |
fix(post-quant): handle sources that keep mtp.* inside a numbered shard
post_quant assumed the source ships a standalone model-mtp.safetensors, which is how JonathanColetti's grafted head is packaged. MuXodious/absolute-heresy is an unmodified full checkpoint, so its mtp.* lives in model-00012-of-00012 -- the copy silently did nothing while the index was still rewritten to point at model-mtp.safetensors, leaving 15 unresolvable tensors. Tensor counts looked correct; the checkpoint would have failed at load. The existing FAILED-CHECKS assertion caught it, which is the design working. Now extracts from the numbered shard when the standalone file is absent. Verified on the heresy build: 1968 tensors, all resolvable, 15 mtp, 333 visual, no missing shards, no orphans. |
||
|
vh
|
74f596b1d3 |
feat(gen-seat): mixed NVFP4+FP8 requant — +18% decode at equal MTP acceptance
Re-quantizes the fleet `gen` seat from weight-only NVFP4A16 to a mixed-precision build: NVFP4 W4A4 for layers 0-55 MLPs, FP8 W8A8 for the attention projections / linear_attn / lm_head / layers 56-63 MLPs, FP8 KV cache. Replicates the scheme of unsloth/Qwen3.8-27B-NVFP4 on the abliterated weights. The queued task named this "W4A8" (NVFP4 weights + FP8 activations). That checkpoint cannot be served: vLLM 0.24's compressed-tensors dispatcher (compressed_tensors.py:704-713) accepts NVFP4 weights with either no input quantization (W4A16, which forces the Marlin kernel) or NVFP4 input quantization (W4A4) -- anything else, FP8 included, raises ValueError at load. CompressedTensorsW4A8Fp8 is INT4 weights gated on an exact-sm90 check, so it is closed on Blackwell twice over. The ~20% intuition was correct; the scheme name was not. Getting FP8 into the mix has to be done per-layer-group. Established the gain before spending GPU time: unsloth's build was already on-box, so serving it as a probe measured +19.1% over our seat at identical MTP acceptance -- a kernel-level result, no requant needed to learn it. Measured, cache-busted, bs=1: decode 80.12 -> 94.53 tok/s (+18.0%) MTP acceptance 47.8% -> 47.7% (unchanged) perplexity (n=6) 6.941 -> 7.059 (+1.7%) abliteration 4/4 -> 4/4 (preserved) weights on disk 27.7 -> 22.5 GB (-19%) Surface test green on the live seat: plain chat, vision, tool calling, thinking split, 36K-token needle retrieval, streaming. All 7 LiteLLM aliases verified routing. GEN_GPU_MEM_UTIL 0.45 -> 0.43: the new weights are 5.2 GB smaller, and at 0.45 the seat absorbed that slack as KV, leaving meromero-charrp 0.18 GiB short of its budget on the shared GPU0 -- it crash-looped. Handing the space back leaves gen 422K tokens of KV (1.6x its 262K context) and both seats co-resident at 89.8/97.9 GB. Also records two measured negatives so they are not re-chased: GEN_SPEC_TOKENS is already optimal at 3 (swept 2/3/4/5 -> 77.1/80.1/78.7/ 75.9 tok/s), and vLLM's prompt_logprobs are ~uniform while speculative decoding is on, so perplexity must be measured with spec off. Pipeline, acceptance harness and raw measurements land in services/gen-seat-mixed-quant/. Rollback is one .env line; the previous build is untouched at /tank/aimodels/qwen38-27b-uncensored-nvfp4. |