Files

T

History

vh f08b6cbddf stacks(char-rp): compose-ify the two NVFP4 RP seats (MeroMero + Dark-Scarlett)

Replaces the ad-hoc docker-run seats with proper compose stacks on ana-ml2, mirrored here:
- meromero-charrp: G4-MeroMero-v2-31B NVFP4A16, char-rp prose (non-thinking, multimodal,
  vision-enabled), GPU0, 256K @ ~2x. util 0.52 (leaves ~4.6GB GPU0 headroom).
- darkscarlett-charrp-reasoning: Dark-Scarlett-v1.0-27B NVFP4A16 (Qwen wrapper recipe),
  char-rp-reasoning thinking seat, GPU1, 256K. MTP deferred (no spec-decode).
Both survive reboot now. Supersede the retired char-rp-gguf + heretic2-charrp-reasoning stacks.

2026-08-12 11:15:33 -07:00

.env.example

stacks(char-rp): compose-ify the two NVFP4 RP seats (MeroMero + Dark-Scarlett)

2026-08-12 11:15:33 -07:00

compose.yaml

stacks(char-rp): compose-ify the two NVFP4 RP seats (MeroMero + Dark-Scarlett)

2026-08-12 11:15:33 -07:00

README.md

stacks(char-rp): compose-ify the two NVFP4 RP seats (MeroMero + Dark-Scarlett)

2026-08-12 11:15:33 -07:00

README.md

darkscarlett-charrp-reasoning — Dark-Scarlett char-rp-reasoning seat (ana-ml2 GPU1)

The thinking RP reasoning seat. Serves the LiteLLM char-rp-reasoning alias.

Model: Dark-Scarlett-v1.0-27B-NVFP4A16-wrapper (ReadyArt Dark-Scarlett-v1.0-27B, Qwen3.6-27B base).
Host/GPU: ana-ml2, GPU1 (co-located with the utility model cluster).
Port: :8018 → LiteLLM char-rp-reasoning.
Context: 256K (--max-model-len 262144). Qwen3.6 hybrid GatedDeltaNet linear-attention → KV-cheap, full 256K easily.
Thinking: default on (--reasoning-parser qwen3; reasoning lands in the reasoning field). ⚠ Consumers need a generous max_tokens or the reasoning eats the whole budget (empty content, finish: length).
Tuning: .env — DS_GPU_MEM_UTIL=0.44, DS_MAX_MODEL_LEN=262144, DS_GPU_ID=1.

Quant note (the wrapper recipe)

Quantized with llm-compressor NVFP4A16 loaded via the Qwen3_5ForConditionalGeneration wrapper class — an AutoModelForCausalLM save produces a flat Qwen3_5TextConfig that both vLLM and SGLang reject; loading the wrapper keeps the config they accept (verified: SGLang errors Qwen3_5ForCausalLM has no SGLang implementation). ModelOpt was blocked by a modelopt↔transformers version deadlock for Qwen3.6. MTP was dropped by the quant loader → deferred → no --speculative-config (spec-decode is net-negative at RP temps anyway). Pipeline: ana-ml2:/tank/aimodels/darkscarlett-nvfp4-work/.

Replaces the retired heretic2-charrp-reasoning (DavidAU Qwen3.6-27B-Heretic2 modelopt NVFP4+MTP) seat.

Deploy

scripts/deploy-stack.sh ana-ml2 darkscarlett-charrp-reasoning
# on host: cp .env.example .env; docker compose up -d