vh
36c173c6a1
Autonomous overnight run under the operator's full-autonomy grant. End state:
fleet up, gen seat untouched, a new verified pen-test seat serving where fable was.
PPL on the orcarouter gen seat (fable downed to free GPU1 for a nospec probe,
probe torn down after): mean 7.07 / median 5.76, within noise of heresy 6.910 /
5.625 and identical to our recipe's usual 7.059. The gen-seat search is settled.
M.O.G.-SEC: chose Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX-BF16 (rev deede677)
over the pre-made ModelOpt NVFP4, which was disqualified on W4A4 4-bit activations
(the AEON degradation mode, catastrophic on a 1M-context model), zero MTP tensors,
and ModelOpt format. Pulled, format-screened (P(<think>) 1.11e-05, clean), quanted
in-house to mixed NVFP4+FP8 (23.4 GB, MTP + vision preserved), and served in the
retired fable slot.
stacks/mog-sec ana-ml2 GPU1 :8019, KV 418,218 tok / 1.60x @ 262K
aliases mog-sec (non-thinking), mog-sec-reasoning (thinking)
gates surface 6/6, MTP 55.3%, format 0/15 leak, vision 7/3/1,
capability 4/4 (delivers offensive-security content)
Served at native 262K, NOT the card's 1M -- the 1M needs YaRN (absent from the
weights' config) plus the SGLang/DFlash2 path the repo ships a deployment kit for,
neither of which is our vLLM surface. A real 1M seat is a separate SGLang project.
Retired char-rp-reasoning + char-rp-fable (zero traffic, pointed at the downed
fable :8019; now 404 cleanly, not repointed -- a security model is not an RP model).
char-rp (meromero) untouched. Vision preprocessor built from the model's own
image_processor block, same trick as the MeroMero seat.
GPU0 seats (gen, meromero) were untouched and healthy throughout. The quant ran in
GPU1 free space with no production seat stopped except fable, which was replaced.
120 lines
6.1 KiB
Markdown
120 lines
6.1 KiB
Markdown
# Gen-seat candidate evaluation — 2026-08-21
|
|
|
|
Cold-Fusion was abandoned (see `persistent-memory.md`); the seat is on
|
|
`qwen38-27b-heresy-nvfp4-mixed`. Two replacement candidates were put up. All facts
|
|
below come from the HF registry and from reading the artifacts directly — the
|
|
safetensors headers were fetched with HTTP **Range** requests, so the tensor census
|
|
cost about a megabyte rather than a 20 GB download.
|
|
|
|
## The candidates
|
|
|
|
| | `orcarouter/Qwen3.8-27B-Uncensored` | `preetpatel/…-NVFP4` |
|
|
|---|---|---|
|
|
| what | BF16 source weights | NVFP4 quant **of orcarouter** |
|
|
| size | 55.6 GB | 19.7 GB |
|
|
| base | `Qwen/Qwen3.8-27B` (**stock Qwen**) | orcarouter |
|
|
| **MTP tensors** | **15 ✓** | **0 ✗** |
|
|
| visual tensors | 333 ✓ | 333 ✓ |
|
|
| scheme | n/a (bf16) | **NVFP4 W4A4** ✗ |
|
|
| `re:^mtp.*` in ignore | n/a | **absent** ✗ |
|
|
| traction | 3,278 dl / 60 likes | 36 dl / 0 likes |
|
|
| gated | yes — **our token already has access** | no |
|
|
| chat template | **sha `c3cf9e34` — byte-identical to the live heresy seat** | same |
|
|
|
|
## Verdict: preetpatel is disqualified, on two independent hard failures
|
|
|
|
**1. Zero MTP tensors.** Read straight from the safetensors header: 2,672 tensors,
|
|
**none** matching `mtp.*`. The author's own `recipe.yaml` asks to ignore
|
|
`re:.*mtp.*`, but the written `config.json` contains no mtp ignore entry at all —
|
|
while `re:.*visual.*` expanded to 110 explicit entries. That asymmetry is the
|
|
signature of llm-compressor pruning an ignore pattern that matched nothing, i.e.
|
|
the MTP head was never loaded and never quantized. It is the same
|
|
`re:^mtp.*`-pruning trap documented in the playbook, seen from the outside.
|
|
|
|
Cost: no speculative decoding. Our seat runs MTP at ~59% acceptance and 118 tok/s;
|
|
without it, roughly half the decode throughput.
|
|
|
|
**2. NVFP4 W4A4 — 4-bit activations.** `input_activations: num_bits 4, type float`.
|
|
This is precisely the AEON failure mode we spent a multi-day saga diagnosing and
|
|
purging: the activation-fidelity gradient is W4A4 < W4+FP8 < W4+bf16, W4A4 was
|
|
responsible for ~15-20% stochastic degeneration, and W4A4 collapses past ~30k
|
|
context. **The gen seat serves 262K.**
|
|
|
|
Either failure alone would rule it out. It is also one day old with 36 downloads.
|
|
|
|
## orcarouter checks out as a quant source
|
|
|
|
Stock-Qwen base (not a reasoning-compression finetune — the trait that sank
|
|
Cold-Fusion), Arditi-et-al. single-direction abliteration, MTP and vision both
|
|
explicitly preserved and verified at 15/333, chat template byte-identical to the
|
|
build we are serving right now, and the gate is already accepted on our token.
|
|
|
|
## Third option, noted and not recommended
|
|
|
|
`orcarouter/Qwen3.8-27B-Uncensored-FP8` — 76,109 downloads, 693 likes, far more
|
|
traction than either candidate. **But 30.9 GB against NVFP4's 22 GB**, and GPU0 is
|
|
zero-sum with meromero co-resident: +9 GB of weights comes straight out of the KV
|
|
pool, taking it from ~14.4 GiB / 403k tokens to roughly 5 GiB / ~150k — which
|
|
breaks 262K context at 1.5x concurrency. Viable only if the seat gives up long
|
|
context or meromero moves.
|
|
|
|
## The imatrix constraint — read before committing to it
|
|
|
|
The operator asked for imatrix if we quant ourselves. **This is not a switch.**
|
|
|
|
`quant_mixed_nvfp4.py` already sets `observer="imatrix_mse"` on the W4A4 group and
|
|
has **never once used it** — llm-compressor logs `no importance data available.
|
|
Falling back to uniform MSE` and proceeds. Playbook §3.13 documents this and warns
|
|
explicitly: *do not "fix" it by assuming an imatrix would help; verify first that
|
|
your llm-compressor version can consume an externally supplied importance matrix at
|
|
all, and in what format.* Parked as `park/…imatrix-mse…` (id 42) with the
|
|
calibration corpus that would feed it.
|
|
|
|
Also note the W4A16 portions of the mixed recipe are **data-free by construction** —
|
|
llm-compressor infers `DataFreePipeline` for weight-only quantization and ignores
|
|
calibration data entirely. Imatrix can only ever bite on the W4A4 MLP group.
|
|
|
|
So "quant with imatrix" is two projects: an unscoped capability investigation, and
|
|
then the ~2h quant. Recommendation is to decouple them — ship the proven recipe
|
|
first, run imatrix as its own bounded experiment. Every A/B we hold is
|
|
uniform-MSE-to-uniform-MSE, so a non-imatrix build stays directly comparable to
|
|
heresy's PPL 6.910 / 47.2% acceptance.
|
|
|
|
## Mandatory step if we pull
|
|
|
|
Run `services/gen-seat-mixed-quant/bench/think-leak/think_prior.py` on the bf16
|
|
**before any GPU time**. It is a ~10s CPU measurement and it is the gate that would
|
|
have disqualified Cold-Fusion before its 300-trial study ever ran. Prior is
|
|
favourable — stock-Qwen base, template identical to heresy, which measures <0.002
|
|
against Cold-Fusion's 0.185 — but measure, don't assume.
|
|
|
|
---
|
|
|
|
# Addendum — M.O.G.-SEC pen-test model (same night)
|
|
|
|
Two `Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX` candidates for the pen-test
|
|
project: a BF16 and a pre-made NVFP4. **Same verdict as gen-seat: pull the BF16,
|
|
quant ourselves.** Read directly off the artifacts via HTTP Range.
|
|
|
|
| | BF16 | pre-made NVFP4 |
|
|
|---|---|---|
|
|
| MTP tensors | 15 ✓ | **0 ✗** |
|
|
| scheme | n/a | **ModelOpt W4A4** ✗ |
|
|
| context | native 262K (config), 1M claimed | same |
|
|
|
|
The pre-made NVFP4 is disqualified on **three** grounds, one unique to this model:
|
|
ModelOpt **W4A4** (4-bit activations — the AEON degradation mode), **zero MTP**,
|
|
and — the sharp one — **W4A4 on a 1M-context model is self-defeating**, since
|
|
W4A4 fidelity collapses past ~30k. A long-context model quanted on the activation
|
|
scheme that fails hardest at long context works against itself.
|
|
|
|
The BF16 quanted cleanly (`mog-sec-27b-nvfp4-mixed`, 23.4 GB) and is **served** in
|
|
the retired fable slot (ana-ml2 GPU1 :8019, aliases `mog-sec` / `mog-sec-reasoning`).
|
|
Gates: format screen 1.11e-05, surface 6/6, MTP 55.3%, vision 7/3/1, and a
|
|
capability smoke 4/4 (it delivers offensive-security content, does not refuse).
|
|
|
|
**The 1M is not real on our path.** `rope_scaling: None` in the weights' config
|
|
(native Qwen3.8 is 262K), and the repo's 1M is an SGLang/DFlash2 deployment kit.
|
|
We serve native 262K. A true 1M seat would be a separate SGLang project — flagged,
|
|
not attempted.
|