stacks(char-rp): compose-ify the two NVFP4 RP seats (MeroMero + Dark-Scarlett)

Replaces the ad-hoc docker-run seats with proper compose stacks on ana-ml2, mirrored here: - meromero-charrp: G4-MeroMero-v2-31B NVFP4A16, char-rp prose (non-thinking, multimodal, vision-enabled), GPU0, 256K @ ~2x. util 0.52 (leaves ~4.6GB GPU0 headroom). - darkscarlett-charrp-reasoning: Dark-Scarlett-v1.0-27B NVFP4A16 (Qwen wrapper recipe), char-rp-reasoning thinking seat, GPU1, 256K. MTP deferred (no spec-decode). Both survive reboot now. Supersede the retired char-rp-gguf + heretic2-charrp-reasoning stacks.
2026-08-12 11:15:33 -07:00
parent 398b58a161
commit f08b6cbddf
6 changed files with 201 additions and 0 deletions
@@ -0,0 +1,4 @@
+# ana-ml2 GPU1 char-rp-reasoning seat (Dark-Scarlett). Real .env lives on the host.
+DS_GPU_MEM_UTIL=0.44
+DS_MAX_MODEL_LEN=262144
+DS_GPU_ID=1
@@ -0,0 +1,28 @@
+# darkscarlett-charrp-reasoning — Dark-Scarlett char-rp-reasoning seat (ana-ml2 GPU1)
+
+The **thinking** RP reasoning seat. Serves the LiteLLM `char-rp-reasoning` alias.
+
+- **Model:** `Dark-Scarlett-v1.0-27B-NVFP4A16-wrapper` (ReadyArt Dark-Scarlett-v1.0-27B, Qwen3.6-27B base).
+- **Host/GPU:** ana-ml2, GPU1 (co-located with the utility model cluster).
+- **Port:** :8018 → LiteLLM `char-rp-reasoning`.
+- **Context:** 256K (`--max-model-len 262144`). Qwen3.6 hybrid GatedDeltaNet linear-attention → KV-cheap, full 256K easily.
+- **Thinking:** default on (`--reasoning-parser qwen3`; reasoning lands in the `reasoning` field). ⚠ Consumers need a generous `max_tokens` or the reasoning eats the whole budget (empty content, `finish: length`).
+- **Tuning:** `.env` — `DS_GPU_MEM_UTIL=0.44`, `DS_MAX_MODEL_LEN=262144`, `DS_GPU_ID=1`.
+
+## Quant note (the wrapper recipe)
+
+Quantized with **llm-compressor NVFP4A16 loaded via the `Qwen3_5ForConditionalGeneration`
+wrapper class** — an `AutoModelForCausalLM` save produces a flat `Qwen3_5TextConfig` that **both
+vLLM and SGLang reject**; loading the wrapper keeps the config they accept (verified: SGLang errors
+`Qwen3_5ForCausalLM has no SGLang implementation`). ModelOpt was blocked by a modelopt↔transformers
+version deadlock for Qwen3.6. **MTP was dropped by the quant loader → deferred → no `--speculative-config`**
+(spec-decode is net-negative at RP temps anyway). Pipeline: `ana-ml2:/tank/aimodels/darkscarlett-nvfp4-work/`.
+
+Replaces the retired **heretic2-charrp-reasoning** (DavidAU Qwen3.6-27B-Heretic2 modelopt NVFP4+MTP) seat.
+
+## Deploy
+
+```bash
+scripts/deploy-stack.sh ana-ml2 darkscarlett-charrp-reasoning
+# on host: cp .env.example .env; docker compose up -d
+```
@@ -0,0 +1,76 @@
+# darkscarlett-charrp-reasoning — Dark-Scarlett-v1.0-27B NVFP4A16 char-rp-reasoning seat
+# (THINKING default) on ana-ml2 GPU1. Replaces the retired heretic2-charrp-reasoning seat.
+#
+# Qwen3.6 hybrid GatedDeltaNet linear-attention -> KV-cheap, serves the full native 256K.
+# Quantized with llm-compressor NVFP4A16 loaded through the Qwen3_5ForConditionalGeneration
+# WRAPPER class (an AutoModelForCausalLM save produces a flat config vLLM/SGLang both reject;
+# the wrapper keeps the config they accept). Served --language-model-only (text RP).
+# MTP was dropped by the quant loader -> deferred -> NO --speculative-config.
+#
+# Co-located on GPU1 with the utility model cluster (rerankers/embed/reward/selene/coder/lfm).
+# Tunables in .env.
+
+name: darkscarlett-charrp-reasoning
+
+services:
+  vllm-darkscarlett-rp:
+    image: ${DS_IMAGE:-vllm/vllm-openai:latest}
+    container_name: ${DS_CONTAINER:-vllm-darkscarlett-rp}
+    restart: unless-stopped
+    ipc: host
+    ports:
+      - "${DS_PORT:-8018}:8000"
+    volumes:
+      - /tank/aimodels:/tank/aimodels
+    environment:
+      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
+      - VLLM_API_KEY=${API_KEY:-}
+    command:
+      - ${DS_MODEL:-/tank/aimodels/darkscarlett-nvfp4-work/Dark-Scarlett-v1.0-27B-NVFP4A16-wrapper}
+      - --quantization
+      - compressed-tensors
+      - --language-model-only
+      - --served-model-name
+      - char-rp-reasoning
+      - --reasoning-parser
+      - qwen3
+      - --tool-call-parser
+      - qwen3_coder
+      - --enable-auto-tool-choice
+      - --max-model-len
+      - "${DS_MAX_MODEL_LEN:-262144}"
+      - --max-num-seqs
+      - "${DS_MAX_NUM_SEQS:-32}"
+      - --gpu-memory-utilization
+      - "${DS_GPU_MEM_UTIL:-0.44}"
+      - --kv-cache-dtype
+      - fp8
+      - --trust-remote-code
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              device_ids:
+                - "${DS_GPU_ID:-1}"
+              capabilities:
+                - gpu
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 600s
+    networks:
+      - tnet
+    labels:
+      - homepage.group=AI - Inference
+      - homepage.name=char-rp-reasoning (Dark-Scarlett NVFP4)
+      - homepage.icon=mdi-brain
+      - homepage.description=Dark-Scarlett-v1.0-27B NVFP4A16 thinking seat, 256K (ana-ml2 GPU1)
+      - homepage.href=http://10.250.50.54:${DS_PORT:-8018}/docs
+
+networks:
+  tnet:
+    name: traefik-net
+    external: true
@@ -0,0 +1,4 @@
+# ana-ml2 GPU0 char-rp prose seat (MeroMero-v2). Real .env lives on the host.
+MEROMERO_GPU_MEM_UTIL=0.52
+MEROMERO_MAX_MODEL_LEN=262144
+MEROMERO_GPU_ID=0
@@ -0,0 +1,20 @@
+# meromero-charrp — MeroMero-v2 char-rp prose seat (ana-ml2 GPU0)
+
+The **non-thinking, multimodal** RP prose seat. Serves the LiteLLM `char-rp` alias.
+
+- **Model:** `G4-MeroMero-v2-31B-NVFP4A16` (Gemma-4-31B, home-quantized weight-only NVFP4A16).
+- **Host/GPU:** ana-ml2, GPU0 (co-located with `gen` / vllm-aeon-gen).
+- **Port:** :8016 → LiteLLM `char-rp`.
+- **Context:** 256K (`--max-model-len 262144`). Gemma-4 uses sliding-window attention → KV-efficient, ~2× concurrency at full context.
+- **Vision:** enabled (image + text). `preprocessor_config.json` was materialized from the model's own `processor_config.json` (`Gemma4ImageProcessor`); audio is config-declared but weightless.
+- **Tuning:** `.env` — `MEROMERO_GPU_MEM_UTIL=0.52` (leaves ~4.6 GB GPU0 headroom), `MEROMERO_MAX_MODEL_LEN=262144`, `MEROMERO_GPU_ID=0`.
+
+Replaces the retired **char-rp-gguf** (Magidonia-24B GGUF / llama.cpp) seat. The quant pipeline
+lives in `ana-ml2:/tank/aimodels/meromero-v2-nvfp4-work/`.
+
+## Deploy
+
+```bash
+scripts/deploy-stack.sh ana-ml2 meromero-charrp     # diffs vs live, prompts y/N
+# on host: cp .env.example .env; docker compose up -d
+```
@@ -0,0 +1,69 @@
+# meromero-charrp — G4-MeroMero-v2-31B NVFP4A16 char-rp PROSE seat (non-thinking, MULTIMODAL)
+# on ana-ml2 GPU0. Replaces the retired Magidonia GGUF seat (char-rp-gguf / llama-charrp).
+#
+# Gemma-4 dense but with SLIDING-WINDOW attention -> KV-efficient, serves the full native 256K
+# at ~2x concurrency. Vision enabled: preprocessor_config.json was materialized from
+# processor_config.json's image_processor section (Gemma4ImageProcessor); audio is
+# config-declared but weightless (no audio tensors) so it serves image + text.
+#
+# Co-located on GPU0 with vllm-aeon-gen (gen). util 0.52 leaves ~5GB GPU0 headroom
+# (0.55 left only ~1.8GB). Tunables in .env.
+
+name: meromero-charrp
+
+services:
+  vllm-meromero-rp:
+    image: ${MEROMERO_IMAGE:-vllm/vllm-openai:latest}
+    container_name: ${MEROMERO_CONTAINER:-vllm-meromero-rp}
+    restart: unless-stopped
+    ipc: host
+    ports:
+      - "${MEROMERO_PORT:-8016}:8000"
+    volumes:
+      - /tank/aimodels:/tank/aimodels
+    environment:
+      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
+      - VLLM_API_KEY=${API_KEY:-}
+    command:
+      - ${MEROMERO_MODEL:-/tank/aimodels/meromero-v2-nvfp4-work/G4-MeroMero-v2-31B-NVFP4A16}
+      - --quantization
+      - compressed-tensors
+      - --served-model-name
+      - char-rp
+      - --max-model-len
+      - "${MEROMERO_MAX_MODEL_LEN:-262144}"
+      - --max-num-seqs
+      - "${MEROMERO_MAX_NUM_SEQS:-32}"
+      - --gpu-memory-utilization
+      - "${MEROMERO_GPU_MEM_UTIL:-0.52}"
+      - --kv-cache-dtype
+      - fp8
+      - --trust-remote-code
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              device_ids:
+                - "${MEROMERO_GPU_ID:-0}"
+              capabilities:
+                - gpu
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 600s
+    networks:
+      - tnet
+    labels:
+      - homepage.group=AI - Inference
+      - homepage.name=char-rp (MeroMero-v2 NVFP4, multimodal)
+      - homepage.icon=mdi-drama-masks
+      - homepage.description=G4-MeroMero-v2-31B NVFP4A16 non-thinking prose seat, vision-enabled, 256K (ana-ml2 GPU0)
+      - homepage.href=http://10.250.50.54:${MEROMERO_PORT:-8016}/docs
+
+networks:
+  tnet:
+    name: traefik-net
+    external: true