VuongKB/ATLAS/Infrastructure/Infrastructure-PFI-Hardware-Profile-ana-ml2.md

created, id, keywords, links, modified, summary, tags, title

created	id	keywords	links	modified	summary	tags	title
2026-04-11T06:15:14+00:00	65e03d42-d7cb-4404-9b55-38b72d37d51f	ana-ml2 10.250.50.54 bare metal RTX 6000 Ada EPYC 9254 PFI-ANA inference server GPU CUDA 12.8 hardware profile		2026-04-11T06:15:14+00:00	Definitive hardware profile for ana-ml2, the bare metal inference server at PFI-ANA colo (10.250.50.54). Dual AMD EPYC 9254, 566 GB RAM, 2× NVIDIA RTX 6000 Ada (~90 GB VRAM). Profiled 2026-04-10.	infrastructure pfi pfi-ana hardware gpu bare-metal inference ana-ml2	Hardware Profile: ana-ml2 (Bare Metal Inference Server)

Hardware Profile: ana-ml2 (Bare Metal Inference Server)

Host: ana-ml2 IP: 10.250.50.54 Type: Physical server (bare metal) Site: PFI-ANA (Colo) Profile Date: 2026-04-10 Profile Script: Infrastructure-PFI/scripts/inference-hardware-profile.sh

System

Spec	Value
OS	Debian GNU/Linux 13 (trixie)
Kernel	6.12.41+deb13-amd64
Architecture	x86_64
Uptime	~31 weeks

CPU

Spec	Value
Model	AMD EPYC 9254 (2 sockets)
Physical Cores	48 (24 per socket)
Logical Threads	96
Base Clock	1500 MHz
Max Boost	4153 MHz
L3 Cache	256 MiB (8 instances)
NUMA Nodes	2
Scaling Governor	schedutil — ⚠️ recommend switching to performance for consistent inference latency

Memory

Spec	Value
Total RAM	566.5 GB
Available	234.3 GB
Used	332.2 GB (58.6%)
Swap	None
Type	DDR5 (likely DDR5-5600 ECC — dmidecode parsing needs refinement for exact spec)

GPU

2× NVIDIA RTX 6000 Ada Generation (Compute Capability 8.9, Ada Lovelace)

GPU 0

Spec	Value
VRAM Total	46,068 MiB (~45 GB, ~48 GB physical minus reserved)
VRAM Used	3,834 MiB (~3.8 GB)
VRAM Free	40,639 MiB (~39.7 GB)
Power Limit	300W / 300W max
Power Draw	Idle (~50W)
Temperature	24°C
SM Clock Max	3105 MHz
Fan	N/A (passive/blower)

GPU 1

Spec	Value
VRAM Total	46,068 MiB (~45 GB)
VRAM Used	4,951 MiB (~4.8 GB)
VRAM Free	39,522 MiB (~38.6 GB)
Power Limit	300W / 300W max
Temperature	25°C
SM Clock Max	3105 MHz

Combined GPU Resources

Metric	Value
Total VRAM	~90 GB (92,136 MiB)
Combined Free VRAM	~80 GB
Total Power Budget	600W

Inferencing Capacity Estimates

Based on ~80 GB combined free VRAM:

Quantization	Max Model Size	Examples
Q4_K_M	~107B parameters	DeepSeek-V2, Mixtral 8x22B
Q8_0	~62B parameters	Llama 3.1 70B (tight fit)
FP16	~41B parameters	Llama 3.1 70B across 2 GPUs

Software Stack

Component	Status
NVIDIA Driver	580.65.06
CUDA Version	12.8
Docker	Present
Python	Present

Known Issues

1. CPU governor — Currently schedutil; switching to performance (cpupower frequency-set -g performance) recommended for inference workloads to reduce latency jitter.
2. Memory type — dmidecode parsing in the profile script returns imprecise data ("2x Correction @ 5600 MT/s"). The RAM is DDR5-5600 ECC but the script needs a parsing fix to capture the exact DIMM type and count accurately.
3. No GPU fan data — RTX 6000 Ada uses blower-style cooling; fan speed field returns N/A on this model.

Relationship to VM-102

VM-102 (PFI-ANA_DOCKER, 10.250.50.70) is a Proxmox virtual machine running on this same bare metal server. It has GPU passthrough configured and hosts the Docker-based inference stack (llama-swap, ComfyUI, ChromaDB, etc.). Hardware resources are shared between the hypervisor and VM-102's passthrough allocations.