docs(ana-ml2): correct GPU spec Ada -> RTX PRO 6000 Blackwell (96GB, cc 12.0)

ana-ml2 was upgraded 2026-06 from dual RTX 6000 Ada (46GB, cc 8.9) to
dual RTX PRO 6000 Blackwell Max-Q (96GB, cc 12.0 / sm_120). Update the
stale hardware facts across the workspace:

- CLAUDE.md servers table row
- servers/ana-ml2/README.md hardware spec (+ refreshed system-details.txt)
- stacks/vllm compose + .env.example FP8/KV comments (Ada cc 8.9 -> Blackwell cc 12.0)
- stacks/llama-swap config VRAM-budget comment (48GB -> 96GB, GPU-0 pin)

Also corrects the adjacent stale 'Phi-4-mini' comment in the granite
service block (the service has been Granite 4.1 8B since 34a43a0).
Doc/comment-only; no runtime change.
This commit is contained in:
2026-06-13 13:36:14 -07:00
parent 0fc9083d16
commit 355a2407a2
6 changed files with 466 additions and 162 deletions
+1 -1
View File
@@ -88,7 +88,7 @@ Observed and standardized across servers:
| Name | IP | Site | Role | Details |
|------|-----|------|------|---------|
| ana-ml2 | 10.250.50.54 | Anaheim (`10.250.0.0/16`) | GPU / AI inference (bare metal, dual RTX 6000 Ada) | `servers/ana-ml2/README.md` |
| ana-ml2 | 10.250.50.54 | Anaheim (`10.250.0.0/16`) | GPU / AI inference (bare metal, dual RTX PRO 6000 Blackwell Max-Q, 96 GB each) | `servers/ana-ml2/README.md` |
| irv-ml1 | 10.100.79.3 (WG) | Irvine — reachable only via WireGuard tunnel from NH3 | GPU / AI inference (bare metal, RTX 3090 + RTX A6000, native stacks) | `servers/irv-ml1/README.md` |
| ana-docker | 10.250.50.70 | Anaheim | General-purpose Docker host (non-GPU VM on pfi-pve) | `servers/ana-docker/README.md` |
| pfi-ana-webhost | 10.250.50.52 | Anaheim | VM on pfi-pve (VMID 110) — web workload | `servers/pfi-ana-webhost/README.md` |
+1 -1
View File
@@ -15,7 +15,7 @@ Primary AI inference host for PFI.
NOT Dell / not the same box as sf-r630 / sfsrv-ana)
- **CPU:** AMD EPYC 9254 24-core (96 threads)
- **RAM:** 566 GB
- **GPUs:** 2x NVIDIA RTX 6000 Ada Generation (46 GB VRAM each, GPU 0 and GPU 1)
- **GPUs:** 2x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB VRAM each, cc 12.0 / sm_120, GPU 0 and GPU 1) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores and doubles VRAM.
- **Storage:** ZFS `zroot` (434 GB root) + `tank` pool (8.6 TB at `/tank`)
- **OS:** Debian 13 (trixie), kernel 6.12.x
- **Docker:** 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2
File diff suppressed because one or more lines are too long
+3 -3
View File
@@ -636,9 +636,9 @@ groups:
#
# Current pins:
# qwen3.5-9b — ~6 GB at Q4 + KV. General-purpose chat baseline.
# VRAM budget: ~6 GB persistent in the pin slot. Single RTX 6000 Ada
# is 48 GB, so this leaves ~40 GB for whichever non-pinned model the
# user invokes alongside.
# VRAM budget: ~6 GB persistent in the pin slot. llama-swap is pinned to
# GPU 0 (a single RTX PRO 6000 Blackwell, 96 GB), so this leaves ~90 GB
# for whichever non-pinned model the user invokes alongside.
#
# granite-4-small WAS pinned here; removed 2026-06-04 — superseded by
# phi4-mini (vLLM FP8, stacks/vllm → vllm-phi4). Freed ~24 GB (120K KV).
+2 -2
View File
@@ -78,7 +78,7 @@ GRANITE_PORT=8004
# pin llama-swap to GPU 0 for clean separation (follow-up).
GRANITE_GPU_ID=1
# Official IBM pre-quantized FP8 (compressed-tensors) — calibrated, ~9.6 GB,
# loaded directly (FP8 native on Ada cc 8.9). Fallback to vLLM-native dynamic FP8
# loaded directly (FP8 native on Blackwell cc 12.0). Fallback to vLLM-native dynamic FP8
# from BF16: GRANITE_MODEL=ibm-granite/granite-4.1-8b + GRANITE_QUANT=fp8.
GRANITE_MODEL=ibm-granite/granite-4.1-8b-fp8
GRANITE_QUANT=compressed-tensors
@@ -90,7 +90,7 @@ GRANITE_SERVED_NAME=granite-4.1-8b
# summarize turn uses ~1k tokens, so the pool holds ~300 concurrently; the "2.33x"
# headline is worst-case (every request maxing 131k). Granite 4.1 supports 131072.
GRANITE_MAX_MODEL_LEN=131072
# FP8 KV cache (native on Ada cc 8.9). At 50K ≈ ~4.2 GB (vs ~8.4 GB at fp16).
# FP8 KV cache (native on Blackwell cc 12.0). At 50K ≈ ~4.2 GB (vs ~8.4 GB at fp16).
GRANITE_KV_CACHE_DTYPE=fp8
# util 0.35 (~33.6 GB) — tuned 2026-06-13 to leave ~3.5 GB free on GPU 1 alongside
# the trio + qwen co-tenants. On this shared card vLLM needs free >= util*total at
+6 -6
View File
@@ -197,10 +197,10 @@ services:
- homepage.description=Skywork-Reward-V2 8B classifier via vLLM (ana-ml2)
- homepage.href=http://10.250.50.54:${REWARD_PORT}/docs
# Phi-4-mini (FP8) — summarizer + "dreaming" agent. Supersedes the
# llama-swap granite-4-small pin. Generative chat model (OpenAI
# /v1/chat/completions), so NO --runner pooling. FP8 on RTX 6000 Ada
# (cc 8.9): near-lossless, ~1.2x, ~6 GB.
# Granite 4.1 8B (FP8) — production summarizer (replaced phi4-mini
# 2026-06-05, which had superseded the llama-swap granite-4-small pin).
# Generative chat model (OpenAI /v1/chat/completions), so NO --runner
# pooling. FP8 on RTX PRO 6000 Blackwell (cc 12.0): near-lossless, ~1.2x, ~6 GB.
vllm-granite:
image: vllm/vllm-openai:${VLLM_VERSION}
container_name: vllm-granite
@@ -218,7 +218,7 @@ services:
command:
# Production summarizer (replaced phi4-mini 2026-06-05). Default = official
# IBM pre-quantized FP8 (compressed-tensors), loaded directly; FP8 is native
# on the RTX 6000 Ada (cc 8.9). Fallback to vLLM-native dynamic FP8 from
# on the RTX PRO 6000 Blackwell (cc 12.0). Fallback to vLLM-native dynamic FP8 from
# BF16: GRANITE_MODEL=ibm-granite/granite-4.1-8b + GRANITE_QUANT=fp8.
- ${GRANITE_MODEL}
- --served-model-name
@@ -240,7 +240,7 @@ services:
# (they were over-provisioned at 5.9x/2.0x/3.9x concurrency); GPU 1 now has
# ~17 GB free after granite, so graph-capture buffers fit. If the trio
# ever grows back, granite may need --enforce-eager again on this card.
# FP8 KV cache — halves KV memory; near-lossless on Ada (cc 8.9).
# FP8 KV cache — halves KV memory; near-lossless on Blackwell (cc 12.0).
- --kv-cache-dtype
- ${GRANITE_KV_CACHE_DTYPE}
# Prefix caching pinned EXPLICIT (vLLM v1 defaults it on, but pin so a