vh/VuongKB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Create: ATLAS/Infrastructure/Infrastructure-PFI-Hardware-Profile-ana-ml2.md

Browse Source
This commit is contained in:
Atlas
2026-04-10 23:15:15 -07:00
parent 28d3513ca7
commit 11dfa9d225
1 changed files with 136 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ATLAS/Infrastructure/Infrastructure-PFI-Hardware-Profile-ana-ml2.md
+136
View File
@@ -0,0 +1,136 @@
---
created: '2026-04-11T06:15:14+00:00'
id: 65e03d42-d7cb-4404-9b55-38b72d37d51f
keywords:
- ana-ml2
- 10.250.50.54
- bare metal
- RTX 6000 Ada
- EPYC 9254
- PFI-ANA
- inference server
- GPU
- CUDA 12.8
- hardware profile
links: []
modified: '2026-04-11T06:15:14+00:00'
summary: Definitive hardware profile for ana-ml2, the bare metal inference server
at PFI-ANA colo (10.250.50.54). Dual AMD EPYC 9254, 566 GB RAM, 2× NVIDIA RTX 6000
Ada (~90 GB VRAM). Profiled 2026-04-10.
tags:
- infrastructure
- pfi
- pfi-ana
- hardware
- gpu
- bare-metal
- inference
- ana-ml2
title: 'Hardware Profile: ana-ml2 (Bare Metal Inference Server)'
---
# Hardware Profile: ana-ml2 (Bare Metal Inference Server)
**Host:** ana-ml2
**IP:** 10.250.50.54
**Type:** Physical server (bare metal)
**Site:** PFI-ANA (Colo)
**Profile Date:** 2026-04-10
**Profile Script:** `Infrastructure-PFI/scripts/inference-hardware-profile.sh`
## System
| Spec | Value |
|---|---|
| **OS** | Debian GNU/Linux 13 (trixie) |
| **Kernel** | 6.12.41+deb13-amd64 |
| **Architecture** | x86_64 |
| **Uptime** | ~31 weeks |
## CPU
| Spec | Value |
|---|---|
| **Model** | AMD EPYC 9254 (2 sockets) |
| **Physical Cores** | 48 (24 per socket) |
| **Logical Threads** | 96 |
| **Base Clock** | 1500 MHz |
| **Max Boost** | 4153 MHz |
| **L3 Cache** | 256 MiB (8 instances) |
| **NUMA Nodes** | 2 |
| **Scaling Governor** | `schedutil` — ⚠️ recommend switching to `performance` for consistent inference latency |
## Memory
| Spec | Value |
|---|---|
| **Total RAM** | 566.5 GB |
| **Available** | 234.3 GB |
| **Used** | 332.2 GB (58.6%) |
| **Swap** | None |
| **Type** | DDR5 (likely DDR5-5600 ECC — dmidecode parsing needs refinement for exact spec) |
## GPU
**2× NVIDIA RTX 6000 Ada Generation** (Compute Capability 8.9, Ada Lovelace)
### GPU 0
| Spec | Value |
|---|---|
| **VRAM Total** | 46,068 MiB (~45 GB, ~48 GB physical minus reserved) |
| **VRAM Used** | 3,834 MiB (~3.8 GB) |
| **VRAM Free** | 40,639 MiB (~39.7 GB) |
| **Power Limit** | 300W / 300W max |
| **Power Draw** | Idle (~50W) |
| **Temperature** | 24°C |
| **SM Clock Max** | 3105 MHz |
| **Fan** | N/A (passive/blower) |
### GPU 1
| Spec | Value |
|---|---|
| **VRAM Total** | 46,068 MiB (~45 GB) |
| **VRAM Used** | 4,951 MiB (~4.8 GB) |
| **VRAM Free** | 39,522 MiB (~38.6 GB) |
| **Power Limit** | 300W / 300W max |
| **Temperature** | 25°C |
| **SM Clock Max** | 3105 MHz |
### Combined GPU Resources
| Metric | Value |
|---|---|
| **Total VRAM** | ~90 GB (92,136 MiB) |
| **Combined Free VRAM** | ~80 GB |
| **Total Power Budget** | 600W |
## Inferencing Capacity Estimates
Based on ~80 GB combined free VRAM:
| Quantization | Max Model Size | Examples |
|---|---|---|
| **Q4_K_M** | ~107B parameters | DeepSeek-V2, Mixtral 8x22B |
| **Q8_0** | ~62B parameters | Llama 3.1 70B (tight fit) |
| **FP16** | ~41B parameters | Llama 3.1 70B across 2 GPUs |
## Software Stack
| Component | Status |
|---|---|
| **NVIDIA Driver** | 580.65.06 |
| **CUDA Version** | 12.8 |
| **Docker** | Present |
| **Python** | Present |
## Known Issues
1. **CPU governor** — Currently `schedutil`; switching to `performance` (`cpupower frequency-set -g performance`) recommended for inference workloads to reduce latency jitter.
2. **Memory type** — dmidecode parsing in the profile script returns imprecise data ("2x Correction @ 5600 MT/s"). The RAM is DDR5-5600 ECC but the script needs a parsing fix to capture the exact DIMM type and count accurately.
3. **No GPU fan data** — RTX 6000 Ada uses blower-style cooling; fan speed field returns N/A on this model.
## Relationship to VM-102
VM-102 (PFI-ANA_DOCKER, 10.250.50.70) is a Proxmox virtual machine running on this same bare metal server. It has GPU passthrough configured and hosts the Docker-based inference stack (llama-swap, ComfyUI, ChromaDB, etc.). Hardware resources are shared between the hypervisor and VM-102's passthrough allocations.