279 lines
12 KiB
Markdown
279 lines
12 KiB
Markdown
---
|
||
created: '2026-04-10T17:35:04+00:00'
|
||
id: 850d5cfb-1b10-42cb-a3ad-fe4066c8cb17
|
||
keywords:
|
||
- docker
|
||
- vm
|
||
- full
|
||
- project
|
||
- chromadb
|
||
- models
|
||
- model
|
||
- reference
|
||
- compose.yaml
|
||
- vcpu
|
||
- gb
|
||
- gpu
|
||
- network
|
||
- llama-swap
|
||
- bare-metal
|
||
- 10.250.50.54
|
||
- 10.250.50.70
|
||
- ana-ml2
|
||
- PFI-ANA_DOCKER
|
||
- inference-server
|
||
- VM-102
|
||
- RTX-6000-Ada
|
||
- EPYC-9254
|
||
- hardware-profile
|
||
links: []
|
||
modified: '2026-04-11T06:15:40+00:00'
|
||
summary: Comprehensive architecture reference for the Infrastructure-PFI project covering
|
||
three Tailscale-linked sites, 11 Proxmox VMs at PFI-ANA colo, 6 Docker services
|
||
with GPU workloads, and llama-swap model management across 20 models.
|
||
tags:
|
||
- infrastructure
|
||
- pfi
|
||
- docker
|
||
- proxmox
|
||
- colocation
|
||
- reference
|
||
- hardware-audit
|
||
title: Infrastructure-PFI — Full Project Index & Architecture Reference
|
||
---
|
||
|
||
# Infrastructure-PFI — Full Project Index & Architecture Reference
|
||
|
||
## Overview
|
||
|
||
Infrastructure-PFI is the configuration and documentation project for the PFI network environment. It covers three physical sites interconnected via Tailscale VPN, hosting a comprehensive infrastructure of Proxmox VMs, Docker services, and AI/ML workloads.
|
||
|
||
---
|
||
|
||
## Sites
|
||
|
||
| Site | Network | Hypervisor | Docker Host | Network Gear |
|
||
|---|---|---|---|---|
|
||
| **ESH-Home** | 10.0.0.0/16 | Proxmox VE | VM at 10.0.50.45 | Ubiquiti |
|
||
| **PFI-NH3** | 10.100.0.0/16 | Proxmox VE | Proxmox VM | Ubiquiti |
|
||
| **PFI-ANA (Colo)** | 10.250.0.0/16 | Proxmox VE at 10.250.250.31:8006 | ana-ml2 (bare metal) at 10.250.50.54 · VM 102 (10.250.50.70) | FortiGate |
|
||
|
||
All sites are linked via Tailscale VPN (VM 106 at the colo).
|
||
|
||
### PFI-ANA Colo — Host Reference
|
||
|
||
| Host | IP | Type | Hardware | Role |
|
||
|---|---|---|---|---|
|
||
| **ana-ml2** (Bare Metal) | 10.250.50.54 | Physical server | 2× EPYC 9254 (48c/96t), 566 GB RAM, 2× RTX 6000 Ada (~90 GB VRAM) | Main inferencing server — direct GPU access |
|
||
| **VM 102** (PFI-ANA_DOCKER) | 10.250.50.70 | Proxmox VM | 8 vCPU, 16 GB RAM, 250G disk | Docker host — llama-swap, ComfyUI, VibeVoice, Parakeet, ChromaDB, Dockge |
|
||
| **VM 100** (PFI-ANA-TRUENAS) | — | Proxmox VM | 4 vCPU, 8 GB RAM, 80G disk | NAS/iSCSI storage |
|
||
| **VM 101** (PFI-ANA-DC) | — | Proxmox VM | 12 vCPU, 24 GB RAM, 240G disk | Windows Domain Controller |
|
||
| **VM 104** (PFI-Mongo) | 10.250.50.81 | Proxmox VM | 8 vCPU, 8 GB RAM | MongoDB server |
|
||
| **VM 106** (PFI-Tailscale) | — | Proxmox VM | 8 vCPU, 2 GB RAM | VPN gateway (Tailscale) |
|
||
|
||
The Proxmox hypervisor management interface is at `10.250.250.31:8006`.
|
||
|
||
> **Hardware Profile:** See [Hardware Profile: ana-ml2](ATLAS/Infrastructure/Infrastructure-PFI-Hardware-Profile-ana-ml2.md) for the full bare metal hardware specification (CPU, GPU, memory, software stack, inference capacity estimates).
|
||
|
||
---
|
||
|
||
## File & Directory Map
|
||
|
||
```
|
||
Infrastructure-PFI/
|
||
├── README.md # Project overview, site summary, structure
|
||
├── configs/pfi-ana/
|
||
│ ├── proxmox/ # Raw Proxmox VM configs (11 VMs)
|
||
│ │ ├── vm-100.conf → PFI-ANA-TRUENAS (TrueNAS SCALE, 4 vCPU, 8GB, 80G disk)
|
||
│ │ ├── vm-101.conf → PFI-ANA-DC (Windows 11 DC, 12 vCPU, 24GB, 240G, UEFI+TPM)
|
||
│ │ ├── vm-102.conf → PFI-ANA-Docker (Linux, 8 vCPU, 16GB, 250G — the Docker host)
|
||
│ │ ├── vm-103.conf → PFI-SlaveBot (Windows 10, 8 vCPU, 8GB, 256G)
|
||
│ │ ├── vm-104.conf → PFI-Mongo (Linux, 8 vCPU, 8GB, 256G, MongoDB)
|
||
│ │ ├── vm-105.conf → PFI-Postgres (Debian 11, 16 vCPU, 8GB, 80G, PostgreSQL)
|
||
│ │ ├── vm-106.conf → PFI-Tailscale (Debian 12, 8 vCPU, 2GB, 256G — VPN gateway)
|
||
│ │ ├── vm-107.conf → PFI-Pterodactyl (Debian 11, 8 vCPU, 8GB, 256G — game servers)
|
||
│ │ ├── vm-108.conf → PFI-ANA--DEV (Debian 11, 8 vCPU, 8GB, 120G — dev env)
|
||
│ │ ├── vm-110.conf → PFI-ANA-Webhost (Debian 11, 16 vCPU, 4GB, 250G — web hosting)
|
||
│ │ └── vm-111.conf → pfi-tacticalrmm (Linux, 16 vCPU, 8GB, 256G — RMM)
|
||
│ └── docker/
|
||
│ ├── compose/ # Live Docker Compose stacks (managed by Dockge)
|
||
│ │ ├── llama-swap/compose.yaml → LLM gateway (port 9292, CUDA)
|
||
│ │ ├── dockge/compose.yaml → Stack manager (port 5001)
|
||
│ │ ├── comfyui/compose.yaml → Image gen (port 8188, GPU)
|
||
│ │ ├── chromadb/compose.yaml → Vector DB (port 8000, token auth)
|
||
│ │ ├── vibevoice/compose.yaml → TTS (port 8745, GPU)
|
||
│ │ └── parakeet/compose.yaml → STT (port 8300, GPU, local build)
|
||
│ ├── conf/
|
||
│ │ └── llama-swap/config.yaml # 20 model definitions, 3 groups
|
||
│ └── compose-examples/ # Reference compose files (not live)
|
||
│ ├── llama-swap/docker-compose.yml
|
||
│ └── chromadb/ # ChromaDB with Traefik labels + demo app
|
||
│ ├── docker-compose.yml
|
||
│ ├── docker-compose.demo.yml
|
||
│ ├── Dockerfile.demo
|
||
│ └── scripts/demo.py
|
||
├── docs/pfi-ana/
|
||
│ ├── docker-stack.md # Docker conventions, 6 services, port map, ASCII diagram
|
||
│ ├── proxmox-vms.md # 11 VMs, startup order, resource summary, network notes
|
||
│ ├── chromadb-setup.md # ChromaDB deployment guide, auth, monitoring
|
||
│ ├── model-list # GGUF inventory with ACTIVE/PENDING/REMOVED status
|
||
│ ├── recommended-model-settings.md # Canonical inference settings for 5 model families
|
||
│ └── hardware-profile-ana-ml2.md # ✅ Bare metal hardware spec (EPYC, RTX 6000 Ada, VRAM, etc.)
|
||
├── scripts/
|
||
│ ├── nemotron-query.sh # Bash client for Nemotron via llama-swap API
|
||
│ ├── setup-chromadb.sh # Deploy ChromaDB on VM 102
|
||
│ ├── backup-chromadb.sh # Hot backup with 30-day retention
|
||
│ ├── health-check-chromadb.sh # Health monitoring with alerting
|
||
│ ├── quickstart-chromadb.sh # One-command ChromaDB setup wrapper
|
||
│ └── inference-hardware-profile.sh # Hardware profiler for AI/ML inference (ana-ml2 bare metal 10.250.50.54)
|
||
└── output/ # Generated output and logs (empty)
|
||
```
|
||
|
||
---
|
||
|
||
## Proxmox VM Startup Order
|
||
|
||
Ordered startup is configured to ensure dependencies are met before dependent services start.
|
||
|
||
| Order | VMID | Name | Delay | Onboot |
|
||
|---|---|---|---|---|
|
||
| 2 | 100 | TRUENAS | 120s | No |
|
||
| 3 | 104 | Mongo | 60s | Yes |
|
||
| 3 | 105 | Postgres | 60s | Yes |
|
||
| 4 | 102 | Docker | — | Yes |
|
||
| 5 | 101 | DC (Windows) | 120s | Yes |
|
||
| 10 | 108 | DEV | 60s | No |
|
||
| 20 | 111 | Tactical RMM | — | Yes |
|
||
| 30 | 110 | Webhost | 120s | Yes |
|
||
|
||
**Note:** VMs 103 (SlaveBot), 106 (Tailscale), and 107 (Pterodactyl) start via `onboot` but without ordered delays.
|
||
|
||
---
|
||
|
||
## Docker Services (VM 102: PFI-ANA_DOCKER)
|
||
|
||
Six services run on the Docker host, with most leveraging GPU acceleration for AI workloads.
|
||
|
||
| Service | Port | GPU | Image |
|
||
|---|---|---|---|
|
||
| Dockge | 5001 | No | `louislam/dockge:latest` |
|
||
| llama-swap | 9292→8080 | Yes (nvidia) | `ghcr.io/mostlygeek/llama-swap:cuda` |
|
||
| ComfyUI | 8188 | Yes | `mmartial/comfyui-nvidia-docker` |
|
||
| ChromaDB | 8000 | No | `chromadb/chroma:latest` |
|
||
| VibeVoice | 8745 | Yes | `eworkerinc/vibevoice:latest` |
|
||
| Parakeet STT | 8300→8000 | Yes | `parakeet-stt` (local build) |
|
||
|
||
**Network Note:** All Docker services run on the `traefik-net` external network.
|
||
|
||
---
|
||
|
||
## llama-swap Model Inventory
|
||
|
||
The llama-swap gateway manages **20 GGUF models** across multiple categories:
|
||
|
||
### By Category
|
||
- **Qwen 3.5**: 4 models
|
||
- **Nemotron**: 2 models
|
||
- **Gemma 4**: 2 models
|
||
- **GLM**: 2 models
|
||
- **Skyfall**: 2 models
|
||
- **Coder**: 1 model
|
||
- **Large/Special**: 1 model
|
||
- **Granite**: 2 models
|
||
- **Embeddings**: 2 models
|
||
- **Reranking**: 1 model
|
||
- **Pending**: 1 (Qwen3-Coder-Next download)
|
||
|
||
### Named Groups
|
||
- **high-reasoning**: 4 models (primary reasoning workloads)
|
||
- **heavy-moe**: 4 models (Mixture-of-Expert architectures)
|
||
- **utility**: 3 persistent models (common utility tasks)
|
||
|
||
---
|
||
|
||
## Storage Layout on VM 102
|
||
|
||
| Host Path | Purpose |
|
||
|---|---|
|
||
| `/opt/docker/compose/<service>/` | Per-service Docker Compose files |
|
||
| `/opt/docker/conf/<service>/` | Per-service configuration files |
|
||
| `/tank/` | Large persistent data (AI models, images, voice data) |
|
||
| `/tank/aimodels/llm/` | GGUF model files (mounted as `/models` in llama-swap) |
|
||
| `/tank/aimodels/img/comfy/` | Image generation models for ComfyUI |
|
||
| `/tank/chromadb/` | ChromaDB persistent vector data |
|
||
| `/tank/vibevoice/` | VibeVoice Hugging Face cache, voices, state |
|
||
| `/opt/docker/backups/chromadb/` | ChromaDB backup archives |
|
||
|
||
---
|
||
|
||
## Network Configuration
|
||
|
||
- **VLAN**: All VMs on VLAN 50 via `vmbr0`
|
||
- **NIC Type**:
|
||
- Linux VMs: `virtio`
|
||
- Windows VMs: `e1000`
|
||
- **Docker Network**: Services on external network `traefik-net`
|
||
- **Docker Host IP**: `10.250.50.70` (VM 102 / PFI-ANA_DOCKER)
|
||
- **Bare Metal Inference Server IP**: ana-ml2 at `10.250.50.54` (physical server, direct GPU access)
|
||
|
||
---
|
||
|
||
## Scripts Reference
|
||
|
||
### ChromaDB Scripts (VM 102)
|
||
| Script | Purpose |
|
||
|---|---|
|
||
| `setup-chromadb.sh` | Deploy ChromaDB container on VM 102 |
|
||
| `backup-chromadb.sh` | Hot backup with 30-day retention |
|
||
| `health-check-chromadb.sh` | Health monitoring with alerting |
|
||
| `quickstart-chromadb.sh` | One-command ChromaDB setup wrapper |
|
||
|
||
### Utility Scripts
|
||
| Script | Purpose |
|
||
|---|---|
|
||
| `nemotron-query.sh` | Bash client for querying Nemotron via llama-swap API |
|
||
| `inference-hardware-profile.sh` | Comprehensive hardware profiler for AI/ML inference planning |
|
||
|
||
### inference-hardware-profile.sh
|
||
|
||
**Target Hosts:**
|
||
- **ana-ml2** (bare metal inferencing server) at `10.250.50.54` — primary target (direct GPU access)
|
||
- **VM 102** (PFI-ANA_DOCKER) at `10.250.50.70` — secondary target (GPU passthrough)
|
||
|
||
A comprehensive system hardware profiler designed for AI/ML inference capacity planning. Captures the full hardware picture needed to evaluate and optimize GPU inference workloads.
|
||
|
||
**What it gathers:**
|
||
- System info (OS, kernel, uptime)
|
||
- CPU details (model, cores, threads, clock speed, scaling governor)
|
||
- Memory (total, available, used, swap)
|
||
- Disk storage (all mounts)
|
||
- GPU resources (model, VRAM, utilization, power, temperature, PCIe, clock speeds, throttling, ECC, active processes, MIG mode)
|
||
- Docker container stats (CPU/memory per container)
|
||
- Inference software stack (Python, PyTorch, CUDA, cuDNN, llama.cpp, Ollama, vLLM)
|
||
- Estimated model capacity based on free VRAM (Q4, Q8, FP16)
|
||
- Inference readiness score with actionable recommendations
|
||
|
||
**Output modes:**
|
||
| Flag | Output |
|
||
|---|---|
|
||
| *(default)* | Full colored terminal report |
|
||
| `--json` | Machine-readable JSON |
|
||
| `--quick` | One-line summary |
|
||
| `--gpu-only` | Full report without Docker sections |
|
||
|
||
> **✅ Completed:** Hardware profile captured for ana-ml2 on 2026-04-10. See [Hardware Profile: ana-ml2](ATLAS/Infrastructure/Infrastructure-PFI-Hardware-Profile-ana-ml2.md). Optionally run on VM 102 (10.250.50.70) for passthrough comparison.
|
||
|
||
---
|
||
|
||
## Known Issues & Observations
|
||
|
||
1. **Nemotron Super** uses `--n-gpu-layers 99` instead of 999 — likely configured to reserve VRAM for other tasks
|
||
2. **Kimi K2** only offloads 2 GPU layers — very CPU-heavy and slow inference
|
||
3. **Granite 4 Small** has `ttl: 0` (persistent caching) but is not assigned to the utility group
|
||
4. **Parakeet STT** references `.env` configuration not tracked in version control
|
||
5. **Qwen3-Coder-Next** download may be incomplete — `.incomplete` files present on disk
|
||
6. ~~**Physical GPU model and VRAM undocumented**~~ — **✅ RESOLVED (2026-04-10):** Full hardware profile captured via `inference-hardware-profile.sh`. ana-ml2 has **2× NVIDIA RTX 6000 Ada** (~90 GB combined VRAM) on dual EPYC 9254 with 566 GB RAM. See [Hardware Profile: ana-ml2](ATLAS/Infrastructure/Infrastructure-PFI-Hardware-Profile-ana-ml2.md). Remaining action: run profile on VM 102 (10.250.50.70) to document passthrough allocation.
|
||
|
||
---
|
||
|
||
*This document was generated following a full infrastructure audit. It serves as the primary reference for understanding, maintaining, and extending the PFI network environment.* |