esh-pfi-infrastructure/docs/pfi/llm-seat-catalog.md

# LLM seat catalog — fv-ml1

The durable, curated record of **what each LLM seat IS**: lineage, provenance,
model-card facts, quantization, speculative decoding, and measured speed/depth.

**This file is hand-curated and complements the generated inventory.** The two split
by volatility:

| file | owns | updated by |
|---|---|---|
| [`fv-ml1-gpu-seat-inventory.md`](fv-ml1-gpu-seat-inventory.md) | **live** placement, VRAM, KV tokens, concurrency, quant format, gateway aliases | `scripts/seat-inventory.py` (read-only, auto) |
| **this file** | **durable** lineage, provenance, cards, licenses, measured tok/s + depth results, "why this seat" | **by hand**, on seat change or re-bench |

Where they overlap (ctx, concurrency, VRAM), the **inventory is authoritative for the
live number**; the values here are point-in-time and dated. If they disagree, the
inventory won and this file is stale — fix it.

## Keeping this current

Update this file whenever a seat changes — model swap, quant change, context/KV edit,
or a new seat. Two commands regenerate the inputs:

```bash
scripts/seat-inventory.py            # live placement/KV/concurrency (docs/pfi/fv-ml1-gpu-seat-inventory.md)
scripts/seat-bench.py                # warm tok/s + deep-prefill OOM check (serial; prints the numbers below)
```

⚠ **Speed and depth numbers are measurements, not facts about the weights** — they
carry a date and a harness. Re-run `seat-bench.py` after any context/KV/quant/spec
change and update the table with the new date. A number without its harness is not a
result (see the measurement-discipline note at the bottom).

---

## Summary (measured 2026-09-14, harness below)

| seat | GPU | model | ctx | conc. | warm tok/s | VRAM | depth verified | OOM |
|---|---|---|---|---|---|---|---|---|
| **cyberprev** (sec) | 0 | Qwen3.8-27B cyber-SFT (abliterated base) | 262,144 | 1.37× | 136.6 | 47.1 GiB | 259,722 tok | none |
| **gen-small** | 0 | Qwen3.6-35B-A3B Heretic | 262,144 | 2.56× | 254.8 | 36.1 GiB | 254,526 tok | none |
| **gen** (flash-next) | 2 | Qwen3.8-Flash-Next (orcarouter) | 262,144 | 1.31× | 170.7 | 95.3 GiB | 254,273 tok | none |
| **char-rp** | 1 | G4-MeroMero-v2-31B (Gemma4 dense) | 262,144 | 1.22× | 62.7 | 42.3 GiB | 254,858 tok | none |
| **char-rp-fast** | 1 | G4-MeroMero-26B-A4B (Gemma4 MoE) | 262,144 | 2.04× | 225.2 | 27.0 GiB | 254,823 tok | none |
| **coder** | 1 | Qwen2.5-Coder-1.5B (base) | 16,384 | 4.70× | 337.3 | 5.6 GiB | 15,905 tok | none |

Support (non-generative): **reward** Skywork-Reward-V2-Llama-3.1-8B (9.2 GiB, GPU1),
**embed** Qwen3-Embedding-0.6B (3.4 GiB, GPU1), **rerank** bge-reranker-v2-m3
(2.1 GiB, GPU1). **gen (flash-next) is on GPU2 and off-limits to rearrangement;
GPU3 is reserved scratch (empty).**

---

## Seats

### cyberprev — the `sec` / `sec-reasoning` seat (GPU 0, :8025)

- **Serves gateway:** `sec`, `sec-reasoning`. Displaced mog-sec 2026-09-14.
- **Lineage:** `Qwen/Qwen3.8-27B` → `hotdogs/Qwen3.8-27B-abliterated` (abliterated base)
  → **offensive-security tool-calling LoRA** trained on `hotdogs/cyber-sft-agent-qwen38`
  (8,400 rows, 22 pentest tools: nmap/sqlmap/metasploit/hydra/…), merged @ scale 1.0
  = `hotdogs/Qwen3.8-27B-abliterated-cyber-preview` → **in-house name-repair** (the raw
  export shipped 850/1199 body tensors with a tripled `language_model.` prefix — an
  unsloth export bug, see [[reference_unsloth_tripled_prefix_export_bug]]) → **house
  mixed NVFP4 quant** → `/tank/aimodels/cyberprev-nvfp4-mixed-v2`.
- **It is a cyber SFT finetune**, NOT "an abliteration" — the abliteration is inherited
  from the base; the new capability is the cyber tool-calling SFT. Card: tool-call
  format 0/6→6/6, correct tool 0/6→6/6, general capability held (KL 0.04 general /
  0.81 tool = surgical re-target). "Will not refuse" per its card.
- **Quant:** compressed-tensors mixed — NVFP4 W4A4 on MLP layers 0-55, FP8 W8A8 on
  attn/linear_attn/lm_head/MLP 56-63, FP8 KV. 15 MTP tensors grafted (bf16), vision
  tower + `re:^mtp.*` in ignore. Pipeline: `services/gen-seat-mixed-quant/`.
- **Speculative:** dflash k=7 (drafter `qwen38-27b-dflash2-drafter`), ~2.77 mean accept len.
- **Context:** native 262,144, **depth-probed clean to 259,722 tokens** (non-repeating
  prompt). Notable: same base arch as the retired mog-sec, which crashed above ~163,840
  on processing depth — cyberprev does not.
- **License:** apache-2.0.

### gen-small — the fast A3B tier (GPU 0, :8026)

- **Serves gateway:** `gen-small`, `gen-small-reasoning`, and backs `summarizer` +
  `classifier` (the bulk/triage aliases). Stood up 2026-09-14.
- **Model:** `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-NVFP4-Experts-Only`
  → `/tank/aimodels/qwen36-35b-a3b-heretic-nvfp4`.
- **Lineage:** `Qwen/Qwen3.6-35B-A3B` (3B-active MoE, 256 experts, hybrid GDN+attn) →
  **Heretic v1.3.0** abliteration (MPOA — Magnitude-Preserving Orthogonal Ablation, the
  house-favorite method) → NVFP4 experts-only quant. **3.8 was requested but there is no
  general Qwen3.8 A3B** (the 3.8 MoEs are Flash-Next and the 2.4T), so this is the 3.6
  fallback.
- **Card:** 88% fewer refusals (10/100 vs 83/100 original) at 0.0015 KL divergence
  (quality preserved). 19 MTP tensors preserved (native).
- **Quant:** modelopt NVFP4, experts-only (256 experts NVFP4; attn/shared-expert/
  linear_attn bf16). Serves as-is, no re-quant.
- **Speculative:** qwen3_5_mtp k=3, **measured 69.6% acceptance / 3.09 mean length** — MTP
  is a clear win here.
- **Context:** native 262,144, depth-clean to 254,526. Cheap KV (A3B + fp8) → 2.56×
  concurrency on only 8 GiB KV.
- **License:** apache-2.0 (per base `Qwen/Qwen3.6-35B-A3B`).
- **Why:** high-volume, low-caliber work (summarization, classification, triage) belongs
  on a fast 3B-active seat, not the premium gen seat.

### gen — flash-next (GPU 2, :8022) — off-limits to rearrangement

- **Serves gateway:** `gen`, `gen-large`, `gen-reasoning`, `summarizer-large`,
  `classifier-large`, `image-judge`, `chat-judge`, `qwen-image-bench`.
- **Model:** `orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4` → **in-house PLE bf16→FP8
  conversion + `layer_types` rename** → `/tank/aimodels/qwen38-flash-next-orcarouter-nvfp4-plefp8`.
  See [[reference_qwen4exp_ple_loader_branch_order]].
- **Lineage:** `Qwen/Qwen3.8-Flash-Next` (176B total: 125B main + 51B n-gram PLE table,
  ~6B active) → orcarouter uncensored NVFP4 → in-house PLE→FP8 so vLLM's
  `Qwen4ExpPLEEmbeddingMethod` selects the FP8 path.
- **Architecture:** the only seat whose weights don't fit its card — the 51B PLE table
  lives in **pinned host RAM**, read over CUDA UVA. GDN linear-attn + QSA hybrid.
- **Quant:** compressed-tensors mixed (W8A16 attn / W4A16 experts) + FP8 PLE.
- **Speculative:** MTP k=3, ~60.4% acceptance.
- **Context:** 262,144, depth-clean to 254,273. Fastest prefill of the fleet (27.7s/254K).

### char-rp — the char quality tier (GPU 1, :8016)

- **Serves gateway:** `char-rp`, `char-rp-reasoning`.
- **Model:** `/tank/aimodels/meromero-v2-nvfp4-work/G4-MeroMero-v2-31B-NVFP4A16` —
  in-house build, **Gemma4 dense, 60 layers**, heretic-abliterated, compressed-tensors
  NVFP4 W4A16.
- **Speed:** 62.7 tok/s warm — the slowest seat, inherent to a dense 31B; this is the
  **quality** tier. Deep prefill 267s/254K (slowest). Gemma4 sliding-window attn
  (window 1024) keeps KV cheap at depth.
- **Context:** 262,144 @ 1.22×, depth-clean to 254,858.

### char-rp-fast — the char speed tier (GPU 1, :8021)

- **Serves gateway:** `char-rp-fast`.
- **Model:** `/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` —
  in-house build, **Gemma4 MoE (26B-A4B)**, heretic-abliterated, compressed-tensors
  NVFP4A16.
- **Speed:** 225.2 tok/s warm — the throughput answer to char-rp's quality. 2.04×
  concurrency (KV pinned ~8.5 GiB). Depth-clean to 254,823.
- **Context:** 262,144 @ 2.04×.

### coder — FIM code-completion (GPU 1, :8020)

- **Serves gateway:** `coder-fast`. Backs Zed edit-predictions.
- **Model:** `Qwen/Qwen2.5-Coder-1.5B` (base, unquantized), fp8 KV.
- **Speed:** 337.3 tok/s (smallest model, fastest). Context 16,384 @ 4.70×.
- **Note:** util-sized; the ~4.7× overshoots the "2-3×" intent because the 1.5B
  weight+overhead floor (~4.2 GiB) sits just under the util knob's resolution. Hitting
  ≤3× reliably needs a `--kv-cache-memory` byte pin (compose change), deferred.

---

## Benchmark harness (state it with any number above)

- **warm decode tok/s:** greedy (temperature 0), **conc=1** (single stream), **n=3**
  reps, median, fixed ~40-word prompt → 300 output tokens. Decode throughput —
  generation is never prefix-cached, so reps are valid; spread was <1% on every seat.
- **deep prefill / OOM:** one **non-repeating** random prompt at ~0.97× max-model-len,
  8 output tokens. PASS = returns AND the seat's allocator log shows **no OOM / CUBLAS /
  illegal-memory** across the probe window (verified 2026-09-14, 0 hits on all seats).
- **Measured serially** (one seat at a time) — no cross-seat contention. These are clean,
  uncontended, single-stream **ceilings**; real aggregate throughput under concurrency is
  higher per-GPU and lower per-request. Re-derive with `scripts/seat-bench.py`.