Files

T

vh 355a2407a2 docs(ana-ml2): correct GPU spec Ada -> RTX PRO 6000 Blackwell (96GB, cc 12.0)

ana-ml2 was upgraded 2026-06 from dual RTX 6000 Ada (46GB, cc 8.9) to
dual RTX PRO 6000 Blackwell Max-Q (96GB, cc 12.0 / sm_120). Update the
stale hardware facts across the workspace:

- CLAUDE.md servers table row
- servers/ana-ml2/README.md hardware spec (+ refreshed system-details.txt)
- stacks/vllm compose + .env.example FP8/KV comments (Ada cc 8.9 -> Blackwell cc 12.0)
- stacks/llama-swap config VRAM-budget comment (48GB -> 96GB, GPU-0 pin)

Also corrects the adjacent stale 'Phi-4-mini' comment in the granite
service block (the service has been Granite 4.1 8B since 34a43a0).
Doc/comment-only; no runtime change.

2026-06-13 13:36:14 -07:00

README.md

docs(ana-ml2): correct GPU spec Ada -> RTX PRO 6000 Blackwell (96GB, cc 12.0)

2026-06-13 13:36:14 -07:00

ssh-target

Initial commit: PFI fleet inventory, stacks, tooling, and backup pipeline

2026-04-20 14:29:48 -07:00

system-details.txt

docs(ana-ml2): correct GPU spec Ada -> RTX PRO 6000 Blackwell (96GB, cc 12.0)

2026-06-13 13:36:14 -07:00

README.md

ana-ml2

Primary AI inference host for PFI.

Network

LAN IP: 10.250.50.54 (in-band, OS-side)
BMC (OOB): 10.250.250.50 — Supermicro IPMI web UI at https://10.250.250.50 (homepage card: PFI-ANA-ML2 BMC)
SSH: standard port 22 on 10.250.50.54

Hardware

Chassis: Supermicro mid-range inferencing server (bare metal, NOT Dell / not the same box as sf-r630 / sfsrv-ana)
CPU: AMD EPYC 9254 24-core (96 threads)
RAM: 566 GB
GPUs: 2x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB VRAM each, cc 12.0 / sm_120, GPU 0 and GPU 1) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores and doubles VRAM.
Storage: ZFS zroot (434 GB root) + tank pool (8.6 TB at /tank)
OS: Debian 13 (trixie), kernel 6.12.x
Docker: 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2

Key paths

Path	Purpose
`/opt/docker/compose/<stack>/`	Compose files
`/opt/docker/conf/<stack>/`	Config bind mounts
`/tank/aimodels/huggingface/`	HF cache (267 GB, pre-downloaded models)
`/tank/aimodels/llm/`	Legacy GGUF models (790 GB, referenced by llama-swap as `/models/`)
`/var/lib/docker/`	Docker data (on zroot)

Running stacks

Stack	Port	Notes
llama-swap	9292	GGUF model server via llama.cpp
vllm-embed (Qwen3)	8001	OpenAI-compatible embeddings; part of the `vllm` stack (GPU 1)
vllm-rerank (Qwen3)	8002	OpenAI-compatible reranker; part of the `vllm` stack (GPU 1)
vllm-reward (Skywork)	8003	Skywork-Reward-V2-8B-AWQ classifier; part of the `vllm` stack (GPU 1)
dockge	5001	Docker stack management UI
dozzle-agent	7007	Log agent; reports to the Dozzle hub on ana-docker
beszel-agent	45876	Metrics agent; reports to the Beszel hub on ana-docker

Retired since last README update:

infinity — replaced by the vllm stack (originally vllm-qwen3, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping a transformers build that knew Qwen3.
LibreChat (+ rag_api, vectordb, mongodb, meilisearch) — removed from this host.
searxng — now hosted on ana-docker for the whole fleet.
Residual networks (librechat_default, kokoro-tts-gpu_default) from prior experiments are still present; safe to docker network rm at leisure.

Refresh state

scripts/refresh-server-info.sh ana-ml2

Latest snapshot: system-details.txt (regenerate as needed).

GPU allocation policy

By default, no container is pinned. For predictable performance when multiple GPU workloads run concurrently:

GPU 0: heavy LLM (llama-swap big models).
GPU 1: light services (the three vllm services share this GPU via --gpu-memory-utilization).

Use deploy.resources.reservations.devices[].device_ids: ["<id>"] in compose to pin.