136 lines
3.9 KiB
Markdown
136 lines
3.9 KiB
Markdown
---
|
||
created: '2026-04-11T06:15:14+00:00'
|
||
id: 65e03d42-d7cb-4404-9b55-38b72d37d51f
|
||
keywords:
|
||
- ana-ml2
|
||
- 10.250.50.54
|
||
- bare metal
|
||
- RTX 6000 Ada
|
||
- EPYC 9254
|
||
- PFI-ANA
|
||
- inference server
|
||
- GPU
|
||
- CUDA 12.8
|
||
- hardware profile
|
||
links: []
|
||
modified: '2026-04-11T06:15:14+00:00'
|
||
summary: Definitive hardware profile for ana-ml2, the bare metal inference server
|
||
at PFI-ANA colo (10.250.50.54). Dual AMD EPYC 9254, 566 GB RAM, 2× NVIDIA RTX 6000
|
||
Ada (~90 GB VRAM). Profiled 2026-04-10.
|
||
tags:
|
||
- infrastructure
|
||
- pfi
|
||
- pfi-ana
|
||
- hardware
|
||
- gpu
|
||
- bare-metal
|
||
- inference
|
||
- ana-ml2
|
||
title: 'Hardware Profile: ana-ml2 (Bare Metal Inference Server)'
|
||
---
|
||
|
||
# Hardware Profile: ana-ml2 (Bare Metal Inference Server)
|
||
|
||
**Host:** ana-ml2
|
||
**IP:** 10.250.50.54
|
||
**Type:** Physical server (bare metal)
|
||
**Site:** PFI-ANA (Colo)
|
||
**Profile Date:** 2026-04-10
|
||
**Profile Script:** `Infrastructure-PFI/scripts/inference-hardware-profile.sh`
|
||
|
||
## System
|
||
|
||
| Spec | Value |
|
||
|---|---|
|
||
| **OS** | Debian GNU/Linux 13 (trixie) |
|
||
| **Kernel** | 6.12.41+deb13-amd64 |
|
||
| **Architecture** | x86_64 |
|
||
| **Uptime** | ~31 weeks |
|
||
|
||
## CPU
|
||
|
||
| Spec | Value |
|
||
|---|---|
|
||
| **Model** | AMD EPYC 9254 (2 sockets) |
|
||
| **Physical Cores** | 48 (24 per socket) |
|
||
| **Logical Threads** | 96 |
|
||
| **Base Clock** | 1500 MHz |
|
||
| **Max Boost** | 4153 MHz |
|
||
| **L3 Cache** | 256 MiB (8 instances) |
|
||
| **NUMA Nodes** | 2 |
|
||
| **Scaling Governor** | `schedutil` — ⚠️ recommend switching to `performance` for consistent inference latency |
|
||
|
||
## Memory
|
||
|
||
| Spec | Value |
|
||
|---|---|
|
||
| **Total RAM** | 566.5 GB |
|
||
| **Available** | 234.3 GB |
|
||
| **Used** | 332.2 GB (58.6%) |
|
||
| **Swap** | None |
|
||
| **Type** | DDR5 (likely DDR5-5600 ECC — dmidecode parsing needs refinement for exact spec) |
|
||
|
||
## GPU
|
||
|
||
**2× NVIDIA RTX 6000 Ada Generation** (Compute Capability 8.9, Ada Lovelace)
|
||
|
||
### GPU 0
|
||
|
||
| Spec | Value |
|
||
|---|---|
|
||
| **VRAM Total** | 46,068 MiB (~45 GB, ~48 GB physical minus reserved) |
|
||
| **VRAM Used** | 3,834 MiB (~3.8 GB) |
|
||
| **VRAM Free** | 40,639 MiB (~39.7 GB) |
|
||
| **Power Limit** | 300W / 300W max |
|
||
| **Power Draw** | Idle (~50W) |
|
||
| **Temperature** | 24°C |
|
||
| **SM Clock Max** | 3105 MHz |
|
||
| **Fan** | N/A (passive/blower) |
|
||
|
||
### GPU 1
|
||
|
||
| Spec | Value |
|
||
|---|---|
|
||
| **VRAM Total** | 46,068 MiB (~45 GB) |
|
||
| **VRAM Used** | 4,951 MiB (~4.8 GB) |
|
||
| **VRAM Free** | 39,522 MiB (~38.6 GB) |
|
||
| **Power Limit** | 300W / 300W max |
|
||
| **Temperature** | 25°C |
|
||
| **SM Clock Max** | 3105 MHz |
|
||
|
||
### Combined GPU Resources
|
||
|
||
| Metric | Value |
|
||
|---|---|
|
||
| **Total VRAM** | ~90 GB (92,136 MiB) |
|
||
| **Combined Free VRAM** | ~80 GB |
|
||
| **Total Power Budget** | 600W |
|
||
|
||
## Inferencing Capacity Estimates
|
||
|
||
Based on ~80 GB combined free VRAM:
|
||
|
||
| Quantization | Max Model Size | Examples |
|
||
|---|---|---|
|
||
| **Q4_K_M** | ~107B parameters | DeepSeek-V2, Mixtral 8x22B |
|
||
| **Q8_0** | ~62B parameters | Llama 3.1 70B (tight fit) |
|
||
| **FP16** | ~41B parameters | Llama 3.1 70B across 2 GPUs |
|
||
|
||
## Software Stack
|
||
|
||
| Component | Status |
|
||
|---|---|
|
||
| **NVIDIA Driver** | 580.65.06 |
|
||
| **CUDA Version** | 12.8 |
|
||
| **Docker** | Present |
|
||
| **Python** | Present |
|
||
|
||
## Known Issues
|
||
|
||
1. **CPU governor** — Currently `schedutil`; switching to `performance` (`cpupower frequency-set -g performance`) recommended for inference workloads to reduce latency jitter.
|
||
2. **Memory type** — dmidecode parsing in the profile script returns imprecise data ("2x Correction @ 5600 MT/s"). The RAM is DDR5-5600 ECC but the script needs a parsing fix to capture the exact DIMM type and count accurately.
|
||
3. **No GPU fan data** — RTX 6000 Ada uses blower-style cooling; fan speed field returns N/A on this model.
|
||
|
||
## Relationship to VM-102
|
||
|
||
VM-102 (PFI-ANA_DOCKER, 10.250.50.70) is a Proxmox virtual machine running on this same bare metal server. It has GPU passthrough configured and hosts the Docker-based inference stack (llama-swap, ComfyUI, ChromaDB, etc.). Hardware resources are shared between the hypervisor and VM-102's passthrough allocations. |