Files

T

History

vh a0fed13801 feat(ana-ml2): replace Qwen3.5-9B vision with Qwen3.6-35B-A3B FP8 on GPU 1

Retire qwen35-vl (Qwen3.5-9B); add qwen36-vl serving the official FP8
Qwen3.6-35B-A3B vision MoE on :8007 under its TRUE name only — no alias.
qwen3.5-9b-fp8 is killed at vLLM AND the litellm gateway (404/400); a model is
never served under a prior model's name. Consumer (comfy-dev/arbo) notified +
migrated; arbo vkeys flipped to all-proxy-models; shared all-agents-local key
repointed to qwen3.6-35b-a3b.

GPU-1 rebalance for the heavier FP8 weights (~34 GB): granite 0.35->0.24 /
131K->64K, embed/rerank 0.05->0.03 (reclaimed util-reservation waste). Verified:
vision correct, 20-concurrent/endpoint load test = no OOM (~7.5 GB headroom).

Drop the llama-swap qwen3.5-9b GPU-0 pin (GPU 0 freed for the creative-writing
hot-swap card). NVFP4 was the lighter fit (~21 GB) but its vLLM ModelOpt-MoE
loader is broken (KeyError w2_input_scale / lm_head.input_scale, vllm #44081);
revisit when fixed.

2026-06-14 14:41:49 -07:00

conf

feat(ana-ml2): replace Qwen3.5-9B vision with Qwen3.6-35B-A3B FP8 on GPU 1

2026-06-14 14:41:49 -07:00

.env.example

Initial commit: PFI fleet inventory, stacks, tooling, and backup pipeline

2026-04-20 14:29:48 -07:00

compose.yaml

feat(llama-swap): pin to GPU 0, reserving it for large-model hot-loads

2026-06-12 15:17:46 -07:00

README.md

vllm: rename stack from vllm-qwen3 → vllm + add Skywork reward classifier

2026-05-13 22:00:26 -07:00

README.md

llama-swap

GGUF model server with on-demand model swapping. Served via llama.cpp's llama-server under the llama-swap proxy.

Server: ana-ml2 Port: 9292 (configurable via .env) GPU: both (unpinned — runtime: nvidia grants access to all devices; per-model GPU selection happens inside config.yaml)

Files

compose.yaml — canonical compose. Deployed to /opt/docker/compose/llama-swap/compose.yaml on ana-ml2.
.env.example — template for the per-host .env. Copy to .env on the server and tweak.
config.yaml — model definitions and groups. Deployed to /opt/docker/conf/llama-swap/config.yaml on the server.

Homepage labels are in the compose file under the AI Systems group, matching the convention used by vllm and infinity.

Deploy a fresh install

scripts/deploy-stack.sh ana-ml2 llama-swap

ssh ana-ml2 '
  cd /opt/docker/compose/llama-swap && \
  cp -n .env.example .env && \
  docker compose config && \
  docker compose up -d && \
  docker compose logs --tail=30
'

Model reference conventions

Modern entries: use -hf <user>/<repo>[:<quant>] — reads from the shared HF cache, nothing to pre-stage outside hf download
Legacy entries: use --model /models/<dir>/<file>.gguf — reads GGUFs from /tank/aimodels/llm/ (pre-HF-cache era, gradually being migrated)

New models should prefer the -hf pattern.

Deploy updates to config only

# After editing config.yaml here:
scp config.yaml ana-ml2:/opt/docker/conf/llama-swap/config.yaml
ssh ana-ml2 'cd /opt/docker/compose/llama-swap && docker compose restart'

Deploy updates to compose only

# After editing compose.yaml or .env.example here:
scripts/deploy-stack.sh ana-ml2 llama-swap
ssh ana-ml2 'cd /opt/docker/compose/llama-swap && docker compose up -d'