docs(ana-ml2): refresh snapshot + sync README to live GPU state

The README's running-stacks table had drifted well behind reality (still listed
llama-swap + only the embed/rerank/reward trio). Regenerated system-details.txt
and rewrote the stacks + GPU-allocation sections from a live docker ps +
nvidia-smi (2026-07-22):

- GPU 0 (hot): vllm-aeon-gen (qwen3.6-35b-a3b-heretic, NVFP4) + vllm-charrp-
  reasoning-nvfp4 (char-rp-reasoning, NVFP4), ~42+45 GB.
- GPU 1 (on-demand): granite-4.1-8b, selene-1-mini-8b, Skywork reward,
  Qwen3 embed/rerank, and the Magidonia-24B char-RP GGUF (llama-charrp), ~91 GB.
- Recorded the dormant on-disk stacks and llama-swap's retirement.
This commit is contained in:
2026-07-22 15:24:55 -07:00
parent 603d0ad555
commit 966324c5f1
2 changed files with 1566 additions and 78 deletions
+55 -17
View File
@@ -32,21 +32,51 @@ Primary AI inference host for PFI.
## Running stacks
| Stack | Port | Notes |
|-------|------|-------|
| llama-swap | 9292 | GGUF model server via llama.cpp |
| vllm-embed (Qwen3) | 8001 | OpenAI-compatible embeddings; part of the `vllm` stack (GPU 1) |
| vllm-rerank (Qwen3) | 8002 | OpenAI-compatible reranker; part of the `vllm` stack (GPU 1) |
| vllm-reward (Skywork) | 8003 | Skywork-Reward-V2-8B-AWQ classifier; part of the `vllm` stack (GPU 1) |
| dockge | 5001 | Docker stack management UI |
| dozzle-agent | 7007 | Log agent; reports to the Dozzle hub on ana-docker |
| beszel-agent | 45876 | Metrics agent; reports to the Beszel hub on ana-docker |
Live inventory as of 2026-07-22. Each model is its own compose stack now
(container `vllm-<x>` / `llama-<x>`); the `vllm` stack proper is just the
embed/rerank/reward trio. GPUs are pinned per container via
`deploy.resources.reservations.devices[].device_ids`.
**Retired since last README update:**
**GPU 0 — heavy RP / reasoning seats (~88/98 GB, hot serving path):**
| Container | Port | Served model | Quant | Ctx |
|-----------|------|--------------|-------|-----|
| `vllm-aeon-gen` | 8015 | `qwen3.6-35b-a3b-heretic` — the "gen" hero seat | NVFP4 (modelopt) | 256k |
| `vllm-charrp-reasoning-nvfp4` | 8018 | `char-rp-reasoning` (R36 reasoning RP) | NVFP4 (modelopt) | 256k |
**GPU 1 — light / eval / retrieval + char-RP GGUF (~91/98 GB, on-demand):**
| Container | Port | Served model | Quant | Ctx |
|-----------|------|--------------|-------|-----|
| `vllm-granite` | 8004 | `granite-4.1-8b` — fleet summarizer/classifier | FP8 (compressed-tensors) | 131k |
| `llama-charrp` | 8016 | `Magidonia-24B-v4.3` Q6_K — char-RP (llama.cpp) | GGUF Q6_K | — |
| `vllm-selene` | 8011 | `selene-1-mini-8b` — Atla LLM-as-judge | FP8 | 32k |
| `vllm-reward` | 8003 | `Skywork-Reward-V2-Llama-3.1-8B-AWQ` — reward classifier | AWQ | 16k |
| `vllm-embed` | 8001 | `Qwen3-Embedding-0.6B` | — | 8k |
| `vllm-rerank` | 8002 | `Qwen3-Reranker-0.6B` | — | 8k |
**Infra / non-GPU:**
| Container | Port | Notes |
|-----------|------|-------|
| `dockge` | 5001 | Docker stack management UI |
| `dozzle-agent` | 7007 | Log agent → Dozzle hub on ana-docker |
| `beszel-agent` | 45876 | Metrics agent → Beszel hub on ana-docker |
Both cards run near-full (~710 GB headroom each) — adding a seat means placing
it on the card with room or evicting a dormant one first.
**Dormant (compose present on disk, containers stopped)** — rollback / audition
seats, safe to leave: `mistral-medium-3.5`, `mistral-small-4(-heretic)`,
`ms32-24b-angel`, `qwen3.5-122b`, `qwopus3.5-122b`, `qwen35-vl`, `qwen36-vl`,
`qwen36-27b-aeon`, `qwen-image-bench`, `vibevoice`, `comfyui`, `kokoro`,
`parakeet`, `vllm-qwen3`.
**Retired:**
- `llama-swap` (former GGUF multiplexer on :9292) — replaced by dedicated
per-model seats (e.g. `llama-charrp`); no longer running.
- `infinity` — replaced by the `vllm` stack (originally `vllm-qwen3`, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping a `transformers` build that knew Qwen3.
- `LibreChat (+ rag_api, vectordb, mongodb, meilisearch)` — removed from this host.
- `searxng` — now hosted on ana-docker for the whole fleet.
- Residual networks (`librechat_default`, `kokoro-tts-gpu_default`) from prior experiments are still present; safe to `docker network rm` at leisure.
- `LibreChat (+ rag_api, vectordb, mongodb, meilisearch)`, `searxng` — removed from this host (searxng now on ana-docker fleet-wide).
## Refresh state
@@ -58,9 +88,17 @@ Latest snapshot: `system-details.txt` (regenerate as needed).
## GPU allocation policy
By default, no container is pinned. For predictable performance when multiple GPU workloads run concurrently:
Every seat is explicitly pinned via `device_ids` (no unpinned containers), and
both cards run ~90% full:
- **GPU 0:** heavy LLM (llama-swap big models).
- **GPU 1:** light services (the three `vllm` services share this GPU via `--gpu-memory-utilization`).
- **GPU 0:** the two heavy NVFP4 seats — `vllm-aeon-gen` (gen) and
`vllm-charrp-reasoning-nvfp4`. The live serving path (near-100% util under
load), ~42 + 45 GB.
- **GPU 1:** everything else — summarizer (granite), judge (selene), reward,
embed, rerank, and the Magidonia char-RP GGUF seat. Bursty/on-demand, idle
between calls, ~91 GB resident.
Use `deploy.resources.reservations.devices[].device_ids: ["<id>"]` in compose to pin.
Pin with `deploy.resources.reservations.devices[].device_ids: ["<id>"]` in
compose. Each service caps its share with `--gpu-memory-utilization`; with both
cards near-full, placing a new seat means freeing room (evict a dormant one) or
trimming a neighbour's utilization first.
File diff suppressed because one or more lines are too long