vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

gen-seat: deploy Qwen3.8-27B-Uncensored gen seat, rename qwen36-27b-aeon->gen-seat

Browse Source 
- New uncensored gen seat: JonathanColetti/Qwen3.8-27B-Uncensored, in-house NVFP4
  W4A16 (compressed-tensors) + grafted bf16 MTP (config ignore re:^mtp.*), vision-intact,
  262K ctx, MTP n=3 (~42% accept, ~68 tok/s). Replaces the qwen3.6-35b-a3b-heretic MoE.
- Rename compose project qwen36-27b-aeon -> gen-seat, container vllm-aeon-gen -> vllm-gen,
  env vars AEON_GEN_* -> GEN_*; drop the dormant vllm-aeon-rp service.
- litellm: repoint 7 aliases (gen/summarizer/summarizer-large/classifier/image-judge/
  qwen-image-bench -> qwen3.8-27b-uncensored; gen-reasoning -> -thinking).
- servers/ana-ml2/README: refresh the gen hero-seat row.
This commit is contained in:
vh
2026-08-15 00:19:20 -07:00
parent dac4acf0c5
commit 680c30e778
6 changed files with 148 additions and 295 deletions
Download Patch File Download Diff File Expand all files Collapse all files
servers/ana-ml2/README.md
+2 -2
View File
@@ -41,7 +41,7 @@ embed/rerank/reward trio. GPUs are pinned per container via
| Container | Port | Served model | Quant | Ctx |
|-----------|------|--------------|-------|-----|
| `vllm-aeon-gen` | 8015 | `qwen3.6-35b-a3b-heretic` — the "gen" hero seat | NVFP4 (modelopt) | 256k |
| `vllm-gen` (project `gen-seat`) | 8015 | `qwen3.8-27b-uncensored` — the "gen" hero seat (Qwen3.8-27B Heretic-abliterated, in-house NVFP4 W4A16 + grafted MTP) | NVFP4 W4A16 (compressed-tensors) | 262k |
| `vllm-charrp-reasoning-nvfp4` | 8018 | `char-rp-reasoning` (R36 reasoning RP) | NVFP4 (modelopt) | 256k |
**GPU 1 — light / eval / retrieval + char-RP GGUF (~91/98 GB, on-demand):**
@@ -91,7 +91,7 @@ Latest snapshot: `system-details.txt` (regenerate as needed).
Every seat is explicitly pinned via `device_ids` (no unpinned containers), and
both cards run ~90% full:
- **GPU 0:** the two heavy NVFP4 seats — `vllm-aeon-gen` (gen) and
- **GPU 0:** the two heavy NVFP4 seats — `vllm-gen` (gen) and
`vllm-charrp-reasoning-nvfp4`. The live serving path (near-100% util under
load), ~42 + 45 GB.
- **GPU 1:** everything else — summarizer (granite), judge (selene), reward,