vh/VuongKB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update: ATLAS/Infrastructure/Infrastructure-PFI-Project-Index.md — Adding inference-hardware-profile.sh to scripts listing and noting that physical GPU model/VRAM are currently undocumented — the new script should be run on VM 102 to capture this gap.

Browse Source
This commit is contained in:
Atlas
2026-04-10 18:28:16 -07:00
parent 8d1fd0c021
commit f45c5dcbd3
1 changed files with 63 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ATLAS/Infrastructure/Infrastructure-PFI-Project-Index.md
+63 -13
View File
@@ -2,20 +2,23 @@
created: '2026-04-10T17:35:04+00:00'
id: 850d5cfb-1b10-42cb-a3ad-fe4066c8cb17
keywords:
- PFI
- Tailscale VPN
- Proxmox VE
- Docker Compose
- docker
- vm
- full
- project
- chromadb
- models
- model
- reference
- compose.yaml
- vcpu
- gb
- gpu
- network
- llama-swap
- ChromaDB
- ComfyUI
- VibeVoice
- Parakeet STT
- VM startup order
- GPU acceleration
- vector database
- 'yes'
links: []
modified: '2026-04-10T17:35:04+00:00'
modified: '2026-04-11T01:28:16+00:00'
summary: Comprehensive architecture reference for the Infrastructure-PFI project covering
three Tailscale-linked sites, 11 Proxmox VMs at PFI-ANA colo, 6 Docker services
with GPU workloads, and llama-swap model management across 20 models.
@@ -95,7 +98,8 @@ Infrastructure-PFI/
│ ├── setup-chromadb.sh # Deploy ChromaDB on VM 102
│ ├── backup-chromadb.sh # Hot backup with 30-day retention
│ ├── health-check-chromadb.sh # Health monitoring with alerting
│ └── quickstart-chromadb.sh # One-command ChromaDB setup wrapper
│ ├── quickstart-chromadb.sh # One-command ChromaDB setup wrapper
│ └── inference-hardware-profile.sh # ⚠️ NEW — Hardware profiler for AI/ML inference planning (VM 102)
└── output/ # Generated output and logs (empty)
```
@@ -187,6 +191,51 @@ The llama-swap gateway manages **20 GGUF models** across multiple categories:
---
## Scripts Reference
### ChromaDB Scripts (VM 102)
| Script | Purpose |
|---|---|
| `setup-chromadb.sh` | Deploy ChromaDB container on VM 102 |
| `backup-chromadb.sh` | Hot backup with 30-day retention |
| `health-check-chromadb.sh` | Health monitoring with alerting |
| `quickstart-chromadb.sh` | One-command ChromaDB setup wrapper |
### Utility Scripts
| Script | Purpose |
|---|---|
| `nemotron-query.sh` | Bash client for querying Nemotron via llama-swap API |
| `inference-hardware-profile.sh` | **NEW** — Comprehensive hardware profiler for AI/ML inference planning |
### inference-hardware-profile.sh
**Target Host:** VM 102 (ana-ml2) at `10.250.50.54` — PFI-ANA Colo
A comprehensive system hardware profiler designed for AI/ML inference capacity planning. Captures the full hardware picture needed to evaluate and optimize GPU inference workloads.
**What it gathers:**
- System info (OS, kernel, uptime)
- CPU details (model, cores, threads, clock speed, scaling governor)
- Memory (total, available, used, swap)
- Disk storage (all mounts)
- GPU resources (model, VRAM, utilization, power, temperature, PCIe, clock speeds, throttling, ECC, active processes, MIG mode)
- Docker container stats (CPU/memory per container)
- Inference software stack (Python, PyTorch, CUDA, cuDNN, llama.cpp, Ollama, vLLM)
- Estimated model capacity based on free VRAM (Q4, Q8, FP16)
- Inference readiness score with actionable recommendations
**Output modes:**
| Flag | Output |
|---|---|
| *(default)* | Full colored terminal report |
| `--json` | Machine-readable JSON |
| `--quick` | One-line summary |
| `--gpu-only` | Full report without Docker sections |
> **⚠️ Action needed:** This script should be run on VM 102 to capture the **physical GPU model, VRAM capacity, and host hardware specs**, which are currently undocumented in this project. The output (especially `--json`) should be saved to close the hardware documentation gap.
---
## Known Issues & Observations
1. **Nemotron Super** uses `--n-gpu-layers 99` instead of 999 — likely configured to reserve VRAM for other tasks
@@ -194,6 +243,7 @@ The llama-swap gateway manages **20 GGUF models** across multiple categories:
3. **Granite 4 Small** has `ttl: 0` (persistent caching) but is not assigned to the utility group
4. **Parakeet STT** references `.env` configuration not tracked in version control
5. **Qwen3-Coder-Next** download may be incomplete — `.incomplete` files present on disk
6. **Physical GPU model and VRAM undocumented** — the physical GPU passed through to VM 102 (its model, total VRAM, PCIe slot, etc.) is not recorded anywhere in project documentation. Run `inference-hardware-profile.sh --json` on VM 102 and save the output to close this gap.
---