diff --git a/stacks/darkscarlett-charrp-reasoning/.env.example b/stacks/darkscarlett-charrp-reasoning/.env.example new file mode 100644 index 0000000..dce5a23 --- /dev/null +++ b/stacks/darkscarlett-charrp-reasoning/.env.example @@ -0,0 +1,4 @@ +# ana-ml2 GPU1 char-rp-reasoning seat (Dark-Scarlett). Real .env lives on the host. +DS_GPU_MEM_UTIL=0.44 +DS_MAX_MODEL_LEN=262144 +DS_GPU_ID=1 diff --git a/stacks/darkscarlett-charrp-reasoning/README.md b/stacks/darkscarlett-charrp-reasoning/README.md new file mode 100644 index 0000000..a6fc24d --- /dev/null +++ b/stacks/darkscarlett-charrp-reasoning/README.md @@ -0,0 +1,28 @@ +# darkscarlett-charrp-reasoning — Dark-Scarlett char-rp-reasoning seat (ana-ml2 GPU1) + +The **thinking** RP reasoning seat. Serves the LiteLLM `char-rp-reasoning` alias. + +- **Model:** `Dark-Scarlett-v1.0-27B-NVFP4A16-wrapper` (ReadyArt Dark-Scarlett-v1.0-27B, Qwen3.6-27B base). +- **Host/GPU:** ana-ml2, GPU1 (co-located with the utility model cluster). +- **Port:** :8018 → LiteLLM `char-rp-reasoning`. +- **Context:** 256K (`--max-model-len 262144`). Qwen3.6 hybrid GatedDeltaNet linear-attention → KV-cheap, full 256K easily. +- **Thinking:** default on (`--reasoning-parser qwen3`; reasoning lands in the `reasoning` field). ⚠ Consumers need a generous `max_tokens` or the reasoning eats the whole budget (empty content, `finish: length`). +- **Tuning:** `.env` — `DS_GPU_MEM_UTIL=0.44`, `DS_MAX_MODEL_LEN=262144`, `DS_GPU_ID=1`. + +## Quant note (the wrapper recipe) + +Quantized with **llm-compressor NVFP4A16 loaded via the `Qwen3_5ForConditionalGeneration` +wrapper class** — an `AutoModelForCausalLM` save produces a flat `Qwen3_5TextConfig` that **both +vLLM and SGLang reject**; loading the wrapper keeps the config they accept (verified: SGLang errors +`Qwen3_5ForCausalLM has no SGLang implementation`). ModelOpt was blocked by a modelopt↔transformers +version deadlock for Qwen3.6. **MTP was dropped by the quant loader → deferred → no `--speculative-config`** +(spec-decode is net-negative at RP temps anyway). Pipeline: `ana-ml2:/tank/aimodels/darkscarlett-nvfp4-work/`. + +Replaces the retired **heretic2-charrp-reasoning** (DavidAU Qwen3.6-27B-Heretic2 modelopt NVFP4+MTP) seat. + +## Deploy + +```bash +scripts/deploy-stack.sh ana-ml2 darkscarlett-charrp-reasoning +# on host: cp .env.example .env; docker compose up -d +``` diff --git a/stacks/darkscarlett-charrp-reasoning/compose.yaml b/stacks/darkscarlett-charrp-reasoning/compose.yaml new file mode 100644 index 0000000..22b5bff --- /dev/null +++ b/stacks/darkscarlett-charrp-reasoning/compose.yaml @@ -0,0 +1,76 @@ +# darkscarlett-charrp-reasoning — Dark-Scarlett-v1.0-27B NVFP4A16 char-rp-reasoning seat +# (THINKING default) on ana-ml2 GPU1. Replaces the retired heretic2-charrp-reasoning seat. +# +# Qwen3.6 hybrid GatedDeltaNet linear-attention -> KV-cheap, serves the full native 256K. +# Quantized with llm-compressor NVFP4A16 loaded through the Qwen3_5ForConditionalGeneration +# WRAPPER class (an AutoModelForCausalLM save produces a flat config vLLM/SGLang both reject; +# the wrapper keeps the config they accept). Served --language-model-only (text RP). +# MTP was dropped by the quant loader -> deferred -> NO --speculative-config. +# +# Co-located on GPU1 with the utility model cluster (rerankers/embed/reward/selene/coder/lfm). +# Tunables in .env. + +name: darkscarlett-charrp-reasoning + +services: + vllm-darkscarlett-rp: + image: ${DS_IMAGE:-vllm/vllm-openai:latest} + container_name: ${DS_CONTAINER:-vllm-darkscarlett-rp} + restart: unless-stopped + ipc: host + ports: + - "${DS_PORT:-8018}:8000" + volumes: + - /tank/aimodels:/tank/aimodels + environment: + - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True + - VLLM_API_KEY=${API_KEY:-} + command: + - ${DS_MODEL:-/tank/aimodels/darkscarlett-nvfp4-work/Dark-Scarlett-v1.0-27B-NVFP4A16-wrapper} + - --quantization + - compressed-tensors + - --language-model-only + - --served-model-name + - char-rp-reasoning + - --reasoning-parser + - qwen3 + - --tool-call-parser + - qwen3_coder + - --enable-auto-tool-choice + - --max-model-len + - "${DS_MAX_MODEL_LEN:-262144}" + - --max-num-seqs + - "${DS_MAX_NUM_SEQS:-32}" + - --gpu-memory-utilization + - "${DS_GPU_MEM_UTIL:-0.44}" + - --kv-cache-dtype + - fp8 + - --trust-remote-code + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: + - "${DS_GPU_ID:-1}" + capabilities: + - gpu + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8000/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 600s + networks: + - tnet + labels: + - homepage.group=AI - Inference + - homepage.name=char-rp-reasoning (Dark-Scarlett NVFP4) + - homepage.icon=mdi-brain + - homepage.description=Dark-Scarlett-v1.0-27B NVFP4A16 thinking seat, 256K (ana-ml2 GPU1) + - homepage.href=http://10.250.50.54:${DS_PORT:-8018}/docs + +networks: + tnet: + name: traefik-net + external: true diff --git a/stacks/meromero-charrp/.env.example b/stacks/meromero-charrp/.env.example new file mode 100644 index 0000000..5ac8ce2 --- /dev/null +++ b/stacks/meromero-charrp/.env.example @@ -0,0 +1,4 @@ +# ana-ml2 GPU0 char-rp prose seat (MeroMero-v2). Real .env lives on the host. +MEROMERO_GPU_MEM_UTIL=0.52 +MEROMERO_MAX_MODEL_LEN=262144 +MEROMERO_GPU_ID=0 diff --git a/stacks/meromero-charrp/README.md b/stacks/meromero-charrp/README.md new file mode 100644 index 0000000..118a9ea --- /dev/null +++ b/stacks/meromero-charrp/README.md @@ -0,0 +1,20 @@ +# meromero-charrp — MeroMero-v2 char-rp prose seat (ana-ml2 GPU0) + +The **non-thinking, multimodal** RP prose seat. Serves the LiteLLM `char-rp` alias. + +- **Model:** `G4-MeroMero-v2-31B-NVFP4A16` (Gemma-4-31B, home-quantized weight-only NVFP4A16). +- **Host/GPU:** ana-ml2, GPU0 (co-located with `gen` / vllm-aeon-gen). +- **Port:** :8016 → LiteLLM `char-rp`. +- **Context:** 256K (`--max-model-len 262144`). Gemma-4 uses sliding-window attention → KV-efficient, ~2× concurrency at full context. +- **Vision:** enabled (image + text). `preprocessor_config.json` was materialized from the model's own `processor_config.json` (`Gemma4ImageProcessor`); audio is config-declared but weightless. +- **Tuning:** `.env` — `MEROMERO_GPU_MEM_UTIL=0.52` (leaves ~4.6 GB GPU0 headroom), `MEROMERO_MAX_MODEL_LEN=262144`, `MEROMERO_GPU_ID=0`. + +Replaces the retired **char-rp-gguf** (Magidonia-24B GGUF / llama.cpp) seat. The quant pipeline +lives in `ana-ml2:/tank/aimodels/meromero-v2-nvfp4-work/`. + +## Deploy + +```bash +scripts/deploy-stack.sh ana-ml2 meromero-charrp # diffs vs live, prompts y/N +# on host: cp .env.example .env; docker compose up -d +``` diff --git a/stacks/meromero-charrp/compose.yaml b/stacks/meromero-charrp/compose.yaml new file mode 100644 index 0000000..c01abf6 --- /dev/null +++ b/stacks/meromero-charrp/compose.yaml @@ -0,0 +1,69 @@ +# meromero-charrp — G4-MeroMero-v2-31B NVFP4A16 char-rp PROSE seat (non-thinking, MULTIMODAL) +# on ana-ml2 GPU0. Replaces the retired Magidonia GGUF seat (char-rp-gguf / llama-charrp). +# +# Gemma-4 dense but with SLIDING-WINDOW attention -> KV-efficient, serves the full native 256K +# at ~2x concurrency. Vision enabled: preprocessor_config.json was materialized from +# processor_config.json's image_processor section (Gemma4ImageProcessor); audio is +# config-declared but weightless (no audio tensors) so it serves image + text. +# +# Co-located on GPU0 with vllm-aeon-gen (gen). util 0.52 leaves ~5GB GPU0 headroom +# (0.55 left only ~1.8GB). Tunables in .env. + +name: meromero-charrp + +services: + vllm-meromero-rp: + image: ${MEROMERO_IMAGE:-vllm/vllm-openai:latest} + container_name: ${MEROMERO_CONTAINER:-vllm-meromero-rp} + restart: unless-stopped + ipc: host + ports: + - "${MEROMERO_PORT:-8016}:8000" + volumes: + - /tank/aimodels:/tank/aimodels + environment: + - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True + - VLLM_API_KEY=${API_KEY:-} + command: + - ${MEROMERO_MODEL:-/tank/aimodels/meromero-v2-nvfp4-work/G4-MeroMero-v2-31B-NVFP4A16} + - --quantization + - compressed-tensors + - --served-model-name + - char-rp + - --max-model-len + - "${MEROMERO_MAX_MODEL_LEN:-262144}" + - --max-num-seqs + - "${MEROMERO_MAX_NUM_SEQS:-32}" + - --gpu-memory-utilization + - "${MEROMERO_GPU_MEM_UTIL:-0.52}" + - --kv-cache-dtype + - fp8 + - --trust-remote-code + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: + - "${MEROMERO_GPU_ID:-0}" + capabilities: + - gpu + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8000/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 600s + networks: + - tnet + labels: + - homepage.group=AI - Inference + - homepage.name=char-rp (MeroMero-v2 NVFP4, multimodal) + - homepage.icon=mdi-drama-masks + - homepage.description=G4-MeroMero-v2-31B NVFP4A16 non-thinking prose seat, vision-enabled, 256K (ana-ml2 GPU0) + - homepage.href=http://10.250.50.54:${MEROMERO_PORT:-8016}/docs + +networks: + tnet: + name: traefik-net + external: true