esh-pfi-infrastructure/docs/pfi/llm-seat-catalog.md

LLM seat catalog — fv-ml1

The durable, curated record of what each LLM seat IS: lineage, provenance, model-card facts, quantization, speculative decoding, and measured speed/depth.

This file is hand-curated and complements the generated inventory. The two split by volatility:

file	owns	updated by
fv-ml1-gpu-seat-inventory.md	live placement, VRAM, KV tokens, concurrency, quant format, gateway aliases	scripts/seat-inventory.py (read-only, auto)
this file	durable lineage, provenance, cards, licenses, measured tok/s + depth results, "why this seat"	by hand, on seat change or re-bench

Where they overlap (ctx, concurrency, VRAM), the inventory is authoritative for the live number; the values here are point-in-time and dated. If they disagree, the inventory won and this file is stale — fix it.

Keeping this current

Update this file whenever a seat changes — model swap, quant change, context/KV edit, or a new seat. Two commands regenerate the inputs:

scripts/seat-inventory.py            # live placement/KV/concurrency (docs/pfi/fv-ml1-gpu-seat-inventory.md)
scripts/seat-bench.py                # warm tok/s + deep-prefill OOM check (serial; prints the numbers below)

⚠ Speed and depth numbers are measurements, not facts about the weights — they carry a date and a harness. Re-run seat-bench.py after any context/KV/quant/spec change and update the table with the new date. A number without its harness is not a result (see the measurement-discipline note at the bottom).

---

Summary (measured 2026-09-14, harness below)

seat	GPU	model	ctx	conc.	warm tok/s	VRAM	depth verified	OOM
cyberprev (sec)	0	Qwen3.8-27B cyber-SFT (abliterated base)	262,144	1.37×	136.6	47.1 GiB	259,722 tok	none
gen-small	0	Qwen3.6-35B-A3B Heretic	262,144	2.56×	254.8	36.1 GiB	254,526 tok	none
gen (flash-next)	2	Qwen3.8-Flash-Next (orcarouter)	262,144	1.31×	170.7	95.3 GiB	254,273 tok	none
char-rp	1	G4-MeroMero-v2-31B (Gemma4 dense)	262,144	1.22×	62.7	42.3 GiB	254,858 tok	none
char-rp-fast	1	G4-MeroMero-26B-A4B (Gemma4 MoE)	262,144	2.04×	225.2	27.0 GiB	254,823 tok	none
coder	1	Qwen2.5-Coder-1.5B (base)	16,384	4.70×	337.3	5.6 GiB	15,905 tok	none

Support (non-generative): reward Skywork-Reward-V2-Llama-3.1-8B (9.2 GiB, GPU1), embed Qwen3-Embedding-0.6B (3.4 GiB, GPU1), rerank bge-reranker-v2-m3 (2.1 GiB, GPU1). gen (flash-next) is on GPU2 and off-limits to rearrangement; GPU3 is reserved scratch (empty).

---

Seats

cyberprev — the sec / sec-reasoning seat (GPU 0, :8025)

• Serves gateway: sec, sec-reasoning. Displaced mog-sec 2026-09-14.
• Lineage: Qwen/Qwen3.8-27B → hotdogs/Qwen3.8-27B-abliterated (abliterated base) → offensive-security tool-calling LoRA trained on hotdogs/cyber-sft-agent-qwen38 (8,400 rows, 22 pentest tools: nmap/sqlmap/metasploit/hydra/…), merged @ scale 1.0 = hotdogs/Qwen3.8-27B-abliterated-cyber-preview → in-house name-repair (the raw export shipped 850/1199 body tensors with a tripled language_model. prefix — an unsloth export bug, see reference_unsloth_tripled_prefix_export_bug) → house mixed NVFP4 quant → /tank/aimodels/cyberprev-nvfp4-mixed-v2.
• It is a cyber SFT finetune, NOT "an abliteration" — the abliteration is inherited from the base; the new capability is the cyber tool-calling SFT. Card: tool-call format 0/6→6/6, correct tool 0/6→6/6, general capability held (KL 0.04 general / 0.81 tool = surgical re-target). "Will not refuse" per its card.
• Quant: compressed-tensors mixed — NVFP4 W4A4 on MLP layers 0-55, FP8 W8A8 on attn/linear_attn/lm_head/MLP 56-63, FP8 KV. 15 MTP tensors grafted (bf16), vision tower + re:^mtp.* in ignore. Pipeline: services/gen-seat-mixed-quant/.
• Speculative: dflash k=7 (drafter qwen38-27b-dflash2-drafter), ~2.77 mean accept len.
• Context: native 262,144, depth-probed clean to 259,722 tokens (non-repeating prompt). Notable: same base arch as the retired mog-sec, which crashed above ~163,840 on processing depth — cyberprev does not.
• License: apache-2.0.

gen-small — the fast A3B tier (GPU 0, :8026)

• Serves gateway: gen-small, gen-small-reasoning, and backs summarizer + classifier (the bulk/triage aliases). Stood up 2026-09-14.
• Model: llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-NVFP4-Experts-Only → /tank/aimodels/qwen36-35b-a3b-heretic-nvfp4.
• Lineage: Qwen/Qwen3.6-35B-A3B (3B-active MoE, 256 experts, hybrid GDN+attn) → Heretic v1.3.0 abliteration (MPOA — Magnitude-Preserving Orthogonal Ablation, the house-favorite method) → NVFP4 experts-only quant. 3.8 was requested but there is no general Qwen3.8 A3B (the 3.8 MoEs are Flash-Next and the 2.4T), so this is the 3.6 fallback.
• Card: 88% fewer refusals (10/100 vs 83/100 original) at 0.0015 KL divergence (quality preserved). 19 MTP tensors preserved (native).
• Quant: modelopt NVFP4, experts-only (256 experts NVFP4; attn/shared-expert/ linear_attn bf16). Serves as-is, no re-quant.
• Speculative: qwen3_5_mtp k=3, measured 69.6% acceptance / 3.09 mean length — MTP is a clear win here.
• Context: native 262,144, depth-clean to 254,526. Cheap KV (A3B + fp8) → 2.56× concurrency on only 8 GiB KV.
• License: apache-2.0 (per base Qwen/Qwen3.6-35B-A3B).
• Why: high-volume, low-caliber work (summarization, classification, triage) belongs on a fast 3B-active seat, not the premium gen seat.

gen — flash-next (GPU 2, :8022) — off-limits to rearrangement

• Serves gateway: gen, gen-large, gen-reasoning, summarizer-large, classifier-large, image-judge, chat-judge, qwen-image-bench.
• Model: orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4 → in-house PLE bf16→FP8 conversion + layer_types rename → /tank/aimodels/qwen38-flash-next-orcarouter-nvfp4-plefp8. See reference_qwen4exp_ple_loader_branch_order.
• Lineage: Qwen/Qwen3.8-Flash-Next (176B total: 125B main + 51B n-gram PLE table, ~6B active) → orcarouter uncensored NVFP4 → in-house PLE→FP8 so vLLM's Qwen4ExpPLEEmbeddingMethod selects the FP8 path.
• Architecture: the only seat whose weights don't fit its card — the 51B PLE table lives in pinned host RAM, read over CUDA UVA. GDN linear-attn + QSA hybrid.
• Quant: compressed-tensors mixed (W8A16 attn / W4A16 experts) + FP8 PLE.
• Speculative: MTP k=3, ~60.4% acceptance.
• Context: 262,144, depth-clean to 254,273. Fastest prefill of the fleet (27.7s/254K).

char-rp — the char quality tier (GPU 1, :8016)

• Serves gateway: char-rp, char-rp-reasoning.
• Model: /tank/aimodels/meromero-v2-nvfp4-work/G4-MeroMero-v2-31B-NVFP4A16 — in-house build, Gemma4 dense, 60 layers, heretic-abliterated, compressed-tensors NVFP4 W4A16.
• Speed: 62.7 tok/s warm — the slowest seat, inherent to a dense 31B; this is the quality tier. Deep prefill 267s/254K (slowest). Gemma4 sliding-window attn (window 1024) keeps KV cheap at depth.
• Context: 262,144 @ 1.22×, depth-clean to 254,858.

char-rp-fast — the char speed tier (GPU 1, :8021)

• Serves gateway: char-rp-fast.
• Model: /tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 — in-house build, Gemma4 MoE (26B-A4B), heretic-abliterated, compressed-tensors NVFP4A16.
• Speed: 225.2 tok/s warm — the throughput answer to char-rp's quality. 2.04× concurrency (KV pinned ~8.5 GiB). Depth-clean to 254,823.
• Context: 262,144 @ 2.04×.

coder — FIM code-completion (GPU 1, :8020)

• Serves gateway: coder-fast. Backs Zed edit-predictions.
• Model: Qwen/Qwen2.5-Coder-1.5B (base, unquantized), fp8 KV.
• Speed: 337.3 tok/s (smallest model, fastest). Context 16,384 @ 4.70×.
• Note: util-sized; the ~4.7× overshoots the "2-3×" intent because the 1.5B weight+overhead floor (~4.2 GiB) sits just under the util knob's resolution. Hitting ≤3× reliably needs a --kv-cache-memory byte pin (compose change), deferred.

---

Benchmark harness (state it with any number above)

• warm decode tok/s: greedy (temperature 0), conc=1 (single stream), n=3 reps, median, fixed ~40-word prompt → 300 output tokens. Decode throughput — generation is never prefix-cached, so reps are valid; spread was <1% on every seat.
• deep prefill / OOM: one non-repeating random prompt at ~0.97× max-model-len, 8 output tokens. PASS = returns AND the seat's allocator log shows no OOM / CUBLAS / illegal-memory across the probe window (verified 2026-09-14, 0 hits on all seats).
• Measured serially (one seat at a time) — no cross-seat contention. These are clean, uncontended, single-stream ceilings; real aggregate throughput under concurrency is higher per-GPU and lower per-request. Re-derive with scripts/seat-bench.py.