vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

stacks(char-rp): compose-ify the two NVFP4 RP seats (MeroMero + Dark-Scarlett)

Browse Source 
Replaces the ad-hoc docker-run seats with proper compose stacks on ana-ml2, mirrored here:
- meromero-charrp: G4-MeroMero-v2-31B NVFP4A16, char-rp prose (non-thinking, multimodal,
  vision-enabled), GPU0, 256K @ ~2x. util 0.52 (leaves ~4.6GB GPU0 headroom).
- darkscarlett-charrp-reasoning: Dark-Scarlett-v1.0-27B NVFP4A16 (Qwen wrapper recipe),
  char-rp-reasoning thinking seat, GPU1, 256K. MTP deferred (no spec-decode).
Both survive reboot now. Supersede the retired char-rp-gguf + heretic2-charrp-reasoning stacks.
This commit is contained in:
vh
2026-08-12 11:15:33 -07:00
parent 398b58a161
commit f08b6cbddf
6 changed files with 201 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/darkscarlett-charrp-reasoning/README.md
+28
View File
@@ -0,0 +1,28 @@
# darkscarlett-charrp-reasoning — Dark-Scarlett char-rp-reasoning seat (ana-ml2 GPU1)
The **thinking** RP reasoning seat. Serves the LiteLLM `char-rp-reasoning` alias.
- **Model:** `Dark-Scarlett-v1.0-27B-NVFP4A16-wrapper` (ReadyArt Dark-Scarlett-v1.0-27B, Qwen3.6-27B base).
- **Host/GPU:** ana-ml2, GPU1 (co-located with the utility model cluster).
- **Port:** :8018 → LiteLLM `char-rp-reasoning`.
- **Context:** 256K (`--max-model-len 262144`). Qwen3.6 hybrid GatedDeltaNet linear-attention → KV-cheap, full 256K easily.
- **Thinking:** default on (`--reasoning-parser qwen3`; reasoning lands in the `reasoning` field). ⚠ Consumers need a generous `max_tokens` or the reasoning eats the whole budget (empty content, `finish: length`).
- **Tuning:** `.env` — `DS_GPU_MEM_UTIL=0.44`, `DS_MAX_MODEL_LEN=262144`, `DS_GPU_ID=1`.
## Quant note (the wrapper recipe)
Quantized with **llm-compressor NVFP4A16 loaded via the `Qwen3_5ForConditionalGeneration`
wrapper class** — an `AutoModelForCausalLM` save produces a flat `Qwen3_5TextConfig` that **both
vLLM and SGLang reject**; loading the wrapper keeps the config they accept (verified: SGLang errors
`Qwen3_5ForCausalLM has no SGLang implementation`). ModelOpt was blocked by a modelopt↔transformers
version deadlock for Qwen3.6. **MTP was dropped by the quant loader → deferred → no `--speculative-config`**
(spec-decode is net-negative at RP temps anyway). Pipeline: `ana-ml2:/tank/aimodels/darkscarlett-nvfp4-work/`.
Replaces the retired **heretic2-charrp-reasoning** (DavidAU Qwen3.6-27B-Heretic2 modelopt NVFP4+MTP) seat.
## Deploy
```bash
scripts/deploy-stack.sh ana-ml2 darkscarlett-charrp-reasoning
# on host: cp .env.example .env; docker compose up -d
```