vh
f08b6cbddf
Replaces the ad-hoc docker-run seats with proper compose stacks on ana-ml2, mirrored here: - meromero-charrp: G4-MeroMero-v2-31B NVFP4A16, char-rp prose (non-thinking, multimodal, vision-enabled), GPU0, 256K @ ~2x. util 0.52 (leaves ~4.6GB GPU0 headroom). - darkscarlett-charrp-reasoning: Dark-Scarlett-v1.0-27B NVFP4A16 (Qwen wrapper recipe), char-rp-reasoning thinking seat, GPU1, 256K. MTP deferred (no spec-decode). Both survive reboot now. Supersede the retired char-rp-gguf + heretic2-charrp-reasoning stacks.
29 lines
1.7 KiB
Markdown
29 lines
1.7 KiB
Markdown
# darkscarlett-charrp-reasoning — Dark-Scarlett char-rp-reasoning seat (ana-ml2 GPU1)
|
|
|
|
The **thinking** RP reasoning seat. Serves the LiteLLM `char-rp-reasoning` alias.
|
|
|
|
- **Model:** `Dark-Scarlett-v1.0-27B-NVFP4A16-wrapper` (ReadyArt Dark-Scarlett-v1.0-27B, Qwen3.6-27B base).
|
|
- **Host/GPU:** ana-ml2, GPU1 (co-located with the utility model cluster).
|
|
- **Port:** :8018 → LiteLLM `char-rp-reasoning`.
|
|
- **Context:** 256K (`--max-model-len 262144`). Qwen3.6 hybrid GatedDeltaNet linear-attention → KV-cheap, full 256K easily.
|
|
- **Thinking:** default on (`--reasoning-parser qwen3`; reasoning lands in the `reasoning` field). ⚠ Consumers need a generous `max_tokens` or the reasoning eats the whole budget (empty content, `finish: length`).
|
|
- **Tuning:** `.env` — `DS_GPU_MEM_UTIL=0.44`, `DS_MAX_MODEL_LEN=262144`, `DS_GPU_ID=1`.
|
|
|
|
## Quant note (the wrapper recipe)
|
|
|
|
Quantized with **llm-compressor NVFP4A16 loaded via the `Qwen3_5ForConditionalGeneration`
|
|
wrapper class** — an `AutoModelForCausalLM` save produces a flat `Qwen3_5TextConfig` that **both
|
|
vLLM and SGLang reject**; loading the wrapper keeps the config they accept (verified: SGLang errors
|
|
`Qwen3_5ForCausalLM has no SGLang implementation`). ModelOpt was blocked by a modelopt↔transformers
|
|
version deadlock for Qwen3.6. **MTP was dropped by the quant loader → deferred → no `--speculative-config`**
|
|
(spec-decode is net-negative at RP temps anyway). Pipeline: `ana-ml2:/tank/aimodels/darkscarlett-nvfp4-work/`.
|
|
|
|
Replaces the retired **heretic2-charrp-reasoning** (DavidAU Qwen3.6-27B-Heretic2 modelopt NVFP4+MTP) seat.
|
|
|
|
## Deploy
|
|
|
|
```bash
|
|
scripts/deploy-stack.sh ana-ml2 darkscarlett-charrp-reasoning
|
|
# on host: cp .env.example .env; docker compose up -d
|
|
```
|