vh
569e1af9ca
Move the ~22-service flat "AI Systems" group off the Main tab into a new four-tab layout (Main / AI / Infrastructure / Toolchain). The AI tab sorts the inference fleet by function into seven groups: AI - Inference gen, char-rp, char-rp-reasoning, Granite summarizer AI - Eval & Retrieval Selene, Skywork Reward, Qwen3 rerank/embed, image-bench AI - Gateways & Chat LiteLLM, Asset Engine, Gateway Chat, Open WebUI, ... AI - Speech (TTS) Chatterbox Fast, Kokoro, mOrpheus AI - Audio Tools Parakeet ASR, YT Voice Clipper AI - Image & Media ComfyUI, Arbo AI - Dormant stopped rollback seats + retired auditions Relabel each stack's homepage.group so canonical stacks/ matches the live containers on ana-ml2, ana-docker, and irv-ml1. Dormant stacks were refreshed with `docker compose up --no-start` so they carry the new label while staying stopped (compose-start rollback preserved). settings.yaml drives tab/order/ columns; services.yaml and README updated to the new scheme.
109 lines
4.3 KiB
YAML
109 lines
4.3 KiB
YAML
# qwopus3.5-122b — OpenYourMind/Qwopus3.5-122B-A10B-Kimi-K2.6-destilled-abliterated-NVFP4
|
|
# on ana-ml2 GPU 0, REPLACING the bjk110 text-only qwen3.5-122b as the `gen` model
|
|
# (operator 2026-06-19: "already ablated, already quanted, vision tower intact").
|
|
#
|
|
# Qwen3.5-122B-A10B MoE, Kimi-K2.6-distilled + abliterated, NVFP4 — and crucially
|
|
# VISION-INTACT (Qwen3_5MoeForConditionalGeneration + vision_config). So it serves as
|
|
# plain MULTIMODAL (no text-only patch, unlike the bjk110 checkpoint which had its
|
|
# vision weights stripped). vLLM carries the arch natively.
|
|
#
|
|
# Served under --served-model-name qwen3.5-122-a10b so the existing litellm records
|
|
# (gen / gen-reasoning / qwen3.5-122-a10b[-reasoning] / qwen-large[-reasoning]) route
|
|
# here UNCHANGED — the operator's "replace those records with this model". The thinking
|
|
# split (chat_template_kwargs.enable_thinking) + tool-calling (qwen3_coder — the
|
|
# OpenYourMind card's specified parser for this checkpoint's XML tool calls).
|
|
#
|
|
# REVERT: down this; the bjk110 qwen3.5-122b stack is still staged.
|
|
# Tunables in .env.
|
|
|
|
name: qwopus3.5-122b
|
|
|
|
services:
|
|
vllm-qwopus35-122b:
|
|
image: ${QWOPUS_IMAGE:-vllm/vllm-openai:latest}
|
|
container_name: ${QWOPUS_CONTAINER_NAME:-vllm-qwopus35-122b}
|
|
restart: unless-stopped
|
|
ipc: host
|
|
ports:
|
|
- "${QWOPUS_PORT:-8013}:8000"
|
|
volumes:
|
|
- /tank/aimodels/huggingface:/hfcache
|
|
- /tank/aimodels/qwopus3.5-122b-nvfp4:/model:ro
|
|
environment:
|
|
- HF_HOME=/hfcache
|
|
- HF_HUB_CACHE=/hfcache/hub
|
|
- VLLM_API_KEY=${API_KEY:-}
|
|
# Reclaims PyTorch's reserved-but-unallocated fragmentation (4.2GB was stranded at
|
|
# util 0.96, starving the FusedMoE workspace → OOM by 0.1GB). Lets the 3.09GB MoE
|
|
# workspace allocate cleanly. Same knob the bjk110 qwen3.5-122b stack ran.
|
|
- PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
|
|
command:
|
|
- /model
|
|
- --served-model-name
|
|
- ${QWOPUS_SERVED_NAME:-qwen3.5-122-a10b}
|
|
- --host
|
|
- 0.0.0.0
|
|
- --port
|
|
- "8000"
|
|
- --gpu-memory-utilization
|
|
- ${QWOPUS_GPU_MEM_UTIL:-0.92}
|
|
- --max-model-len
|
|
- ${QWOPUS_MAX_MODEL_LEN:-131072}
|
|
- --max-num-seqs
|
|
- ${QWOPUS_MAX_NUM_SEQS:-8}
|
|
- --max-num-batched-tokens
|
|
- "32768"
|
|
- --trust-remote-code
|
|
- --dtype
|
|
- auto
|
|
- --enable-prefix-caching
|
|
- --enable-chunked-prefill
|
|
# FULL 256K context on the STABLE image. fp8 KV (near-lossless) measured an 11.8GB
|
|
# pool = 934,600 tokens = 3.5x concurrency at the full 262144 window. CUDA graphs ON
|
|
# (no --enforce-eager) for decode tok/s. BINDING LIMIT = the FusedMoE transient
|
|
# workspace (3.09GB, allocated OUTSIDE vLLM's budget into free VRAM): at util 0.96
|
|
# only 2.99GB was free → OOM by 0.1GB, worsened by 4.2GB PyTorch fragmentation.
|
|
# FIX = expandable_segments (env above, reclaims the fragmentation) + util 0.95 for
|
|
# margin. The card can't go to 0 free — this workspace is the floor. video kept
|
|
# ENABLED (operator wants it; banked at util 0.95 with headroom) — the video encoder
|
|
# profiling eats into the budget so KV concurrency drops some, but stays well above 2x.
|
|
- --kv-cache-dtype
|
|
- ${QWOPUS_KV_CACHE_DTYPE:-fp8}
|
|
- --limit-mm-per-prompt
|
|
- '{"image": 2, "video": 1}'
|
|
# reasoning split + tool-calling. The OpenYourMind card specifies qwen3_coder
|
|
# as the tool-call parser for this checkpoint (Qwen3.5 XML tool-call format).
|
|
- --reasoning-parser
|
|
- qwen3
|
|
- --enable-auto-tool-choice
|
|
- --tool-call-parser
|
|
- qwen3_coder
|
|
deploy:
|
|
resources:
|
|
reservations:
|
|
devices:
|
|
- driver: nvidia
|
|
device_ids:
|
|
- "${QWOPUS_GPU_ID:-0}"
|
|
capabilities:
|
|
- gpu
|
|
healthcheck:
|
|
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 3
|
|
start_period: 900s
|
|
networks:
|
|
- tnet
|
|
labels:
|
|
- homepage.group=AI - Dormant
|
|
- homepage.name=Qwopus3.5-122B-A10B (abliterated NVFP4, vision)
|
|
- homepage.icon=mdi-creation
|
|
- homepage.description=Kimi-distilled abliterated Qwen3.5-122B-A10B NVFP4, vision-intact, the `gen` model (ana-ml2 GPU 0)
|
|
- homepage.href=http://10.250.50.54:${QWOPUS_PORT:-8013}/docs
|
|
|
|
networks:
|
|
tnet:
|
|
name: traefik-net
|
|
external: true
|