273 lines
12 KiB
Markdown
273 lines
12 KiB
Markdown
---
|
|
created: '2026-04-10T17:35:04+00:00'
|
|
id: 850d5cfb-1b10-42cb-a3ad-fe4066c8cb17
|
|
keywords:
|
|
- docker
|
|
- vm
|
|
- full
|
|
- project
|
|
- chromadb
|
|
- models
|
|
- model
|
|
- reference
|
|
- compose.yaml
|
|
- vcpu
|
|
- gb
|
|
- gpu
|
|
- network
|
|
- llama-swap
|
|
- bare-metal
|
|
- 10.250.50.54
|
|
- 10.250.50.70
|
|
- ana-ml2
|
|
- PFI-ANA_DOCKER
|
|
- inference-server
|
|
- VM-102
|
|
links: []
|
|
modified: '2026-04-11T01:33:45+00:00'
|
|
summary: Comprehensive architecture reference for the Infrastructure-PFI project covering
|
|
three Tailscale-linked sites, 11 Proxmox VMs at PFI-ANA colo, 6 Docker services
|
|
with GPU workloads, and llama-swap model management across 20 models.
|
|
tags:
|
|
- infrastructure
|
|
- pfi
|
|
- docker
|
|
- proxmox
|
|
- colocation
|
|
- reference
|
|
- hardware-audit
|
|
title: Infrastructure-PFI — Full Project Index & Architecture Reference
|
|
---
|
|
|
|
# Infrastructure-PFI — Full Project Index & Architecture Reference
|
|
|
|
## Overview
|
|
|
|
Infrastructure-PFI is the configuration and documentation project for the PFI network environment. It covers three physical sites interconnected via Tailscale VPN, hosting a comprehensive infrastructure of Proxmox VMs, Docker services, and AI/ML workloads.
|
|
|
|
---
|
|
|
|
## Sites
|
|
|
|
| Site | Network | Hypervisor | Docker Host | Network Gear |
|
|
|---|---|---|---|---|
|
|
| **ESH-Home** | 10.0.0.0/16 | Proxmox VE | VM at 10.0.50.45 | Ubiquiti |
|
|
| **PFI-NH3** | 10.100.0.0/16 | Proxmox VE | Proxmox VM | Ubiquiti |
|
|
| **PFI-ANA (Colo)** | 10.250.0.0/16 | Proxmox VE at 10.250.250.31:8006 | Bare metal at 10.250.50.54 · VM 102 (10.250.50.70, ana-ml2) | FortiGate |
|
|
|
|
All sites are linked via Tailscale VPN (VM 106 at the colo).
|
|
|
|
### PFI-ANA Colo — Host Reference
|
|
|
|
| Host | IP | Type | Role |
|
|
|---|---|---|---|
|
|
| **Bare Metal** | 10.250.50.54 | Physical server | Main inferencing server — direct GPU access, runs inference workloads |
|
|
| **VM 102** (PFI-ANA_DOCKER) | 10.250.50.70 | Proxmox VM | Docker host — llama-swap, ComfyUI, VibeVoice, Parakeet, ChromaDB, Dockge |
|
|
| **VM 100** (PFI-ANA-TRUENAS) | — | Proxmox VM | NAS/iSCSI storage |
|
|
| **VM 101** (PFI-ANA-DC) | — | Proxmox VM | Windows Domain Controller |
|
|
| **VM 104** (PFI-Mongo) | 10.250.50.81 | Proxmox VM | MongoDB server |
|
|
| **VM 106** (PFI-Tailscale) | — | Proxmox VM | VPN gateway (Tailscale) |
|
|
|
|
The Proxmox hypervisor management interface is at `10.250.250.31:8006`.
|
|
|
|
---
|
|
|
|
## File & Directory Map
|
|
|
|
```
|
|
Infrastructure-PFI/
|
|
├── README.md # Project overview, site summary, structure
|
|
├── configs/pfi-ana/
|
|
│ ├── proxmox/ # Raw Proxmox VM configs (11 VMs)
|
|
│ │ ├── vm-100.conf → PFI-ANA-TRUENAS (TrueNAS SCALE, 4 vCPU, 8GB, 80G disk)
|
|
│ │ ├── vm-101.conf → PFI-ANA-DC (Windows 11 DC, 12 vCPU, 24GB, 240G, UEFI+TPM)
|
|
│ │ ├── vm-102.conf → PFI-ANA-Docker (Linux, 8 vCPU, 16GB, 250G — the Docker host)
|
|
│ │ ├── vm-103.conf → PFI-SlaveBot (Windows 10, 8 vCPU, 8GB, 256G)
|
|
│ │ ├── vm-104.conf → PFI-Mongo (Linux, 8 vCPU, 8GB, 256G, MongoDB)
|
|
│ │ ├── vm-105.conf → PFI-Postgres (Debian 11, 16 vCPU, 8GB, 80G, PostgreSQL)
|
|
│ │ ├── vm-106.conf → PFI-Tailscale (Debian 12, 8 vCPU, 2GB, 256G — VPN gateway)
|
|
│ │ ├── vm-107.conf → PFI-Pterodactyl (Debian 11, 8 vCPU, 8GB, 256G — game servers)
|
|
│ │ ├── vm-108.conf → PFI-ANA--DEV (Debian 11, 8 vCPU, 8GB, 120G — dev env)
|
|
│ │ ├── vm-110.conf → PFI-ANA-Webhost (Debian 11, 16 vCPU, 4GB, 250G — web hosting)
|
|
│ │ └── vm-111.conf → pfi-tacticalrmm (Linux, 16 vCPU, 8GB, 256G — RMM)
|
|
│ └── docker/
|
|
│ ├── compose/ # Live Docker Compose stacks (managed by Dockge)
|
|
│ │ ├── llama-swap/compose.yaml → LLM gateway (port 9292, CUDA)
|
|
│ │ ├── dockge/compose.yaml → Stack manager (port 5001)
|
|
│ │ ├── comfyui/compose.yaml → Image gen (port 8188, GPU)
|
|
│ │ ├── chromadb/compose.yaml → Vector DB (port 8000, token auth)
|
|
│ │ ├── vibevoice/compose.yaml → TTS (port 8745, GPU)
|
|
│ │ └── parakeet/compose.yaml → STT (port 8300, GPU, local build)
|
|
│ ├── conf/
|
|
│ │ └── llama-swap/config.yaml # 20 model definitions, 3 groups
|
|
│ └── compose-examples/ # Reference compose files (not live)
|
|
│ ├── llama-swap/docker-compose.yml
|
|
│ └── chromadb/ # ChromaDB with Traefik labels + demo app
|
|
│ ├── docker-compose.yml
|
|
│ ├── docker-compose.demo.yml
|
|
│ ├── Dockerfile.demo
|
|
│ └── scripts/demo.py
|
|
├── docs/pfi-ana/
|
|
│ ├── docker-stack.md # Docker conventions, 6 services, port map, ASCII diagram
|
|
│ ├── proxmox-vms.md # 11 VMs, startup order, resource summary, network notes
|
|
│ ├── chromadb-setup.md # ChromaDB deployment guide, auth, monitoring
|
|
│ ├── model-list # GGUF inventory with ACTIVE/PENDING/REMOVED status
|
|
│ └── recommended-model-settings.md # Canonical inference settings for 5 model families
|
|
├── scripts/
|
|
│ ├── nemotron-query.sh # Bash client for Nemotron via llama-swap API
|
|
│ ├── setup-chromadb.sh # Deploy ChromaDB on VM 102
|
|
│ ├── backup-chromadb.sh # Hot backup with 30-day retention
|
|
│ ├── health-check-chromadb.sh # Health monitoring with alerting
|
|
│ ├── quickstart-chromadb.sh # One-command ChromaDB setup wrapper
|
|
│ └── inference-hardware-profile.sh # Hardware profiler for AI/ML inference (bare metal 10.250.50.54)
|
|
└── output/ # Generated output and logs (empty)
|
|
```
|
|
|
|
---
|
|
|
|
## Proxmox VM Startup Order
|
|
|
|
Ordered startup is configured to ensure dependencies are met before dependent services start.
|
|
|
|
| Order | VMID | Name | Delay | Onboot |
|
|
|---|---|---|---|---|
|
|
| 2 | 100 | TRUENAS | 120s | No |
|
|
| 3 | 104 | Mongo | 60s | Yes |
|
|
| 3 | 105 | Postgres | 60s | Yes |
|
|
| 4 | 102 | Docker | — | Yes |
|
|
| 5 | 101 | DC (Windows) | 120s | Yes |
|
|
| 10 | 108 | DEV | 60s | No |
|
|
| 20 | 111 | Tactical RMM | — | Yes |
|
|
| 30 | 110 | Webhost | 120s | Yes |
|
|
|
|
**Note:** VMs 103 (SlaveBot), 106 (Tailscale), and 107 (Pterodactyl) start via `onboot` but without ordered delays.
|
|
|
|
---
|
|
|
|
## Docker Services (VM 102: ana-ml2)
|
|
|
|
Six services run on the Docker host, with most leveraging GPU acceleration for AI workloads.
|
|
|
|
| Service | Port | GPU | Image |
|
|
|---|---|---|---|
|
|
| Dockge | 5001 | No | `louislam/dockge:latest` |
|
|
| llama-swap | 9292→8080 | Yes (nvidia) | `ghcr.io/mostlygeek/llama-swap:cuda` |
|
|
| ComfyUI | 8188 | Yes | `mmartial/comfyui-nvidia-docker` |
|
|
| ChromaDB | 8000 | No | `chromadb/chroma:latest` |
|
|
| VibeVoice | 8745 | Yes | `eworkerinc/vibevoice:latest` |
|
|
| Parakeet STT | 8300→8000 | Yes | `parakeet-stt` (local build) |
|
|
|
|
**Network Note:** All Docker services run on the `traefik-net` external network.
|
|
|
|
---
|
|
|
|
## llama-swap Model Inventory
|
|
|
|
The llama-swap gateway manages **20 GGUF models** across multiple categories:
|
|
|
|
### By Category
|
|
- **Qwen 3.5**: 4 models
|
|
- **Nemotron**: 2 models
|
|
- **Gemma 4**: 2 models
|
|
- **GLM**: 2 models
|
|
- **Skyfall**: 2 models
|
|
- **Coder**: 1 model
|
|
- **Large/Special**: 1 model
|
|
- **Granite**: 2 models
|
|
- **Embeddings**: 2 models
|
|
- **Reranking**: 1 model
|
|
- **Pending**: 1 (Qwen3-Coder-Next download)
|
|
|
|
### Named Groups
|
|
- **high-reasoning**: 4 models (primary reasoning workloads)
|
|
- **heavy-moe**: 4 models (Mixture-of-Expert architectures)
|
|
- **utility**: 3 persistent models (common utility tasks)
|
|
|
|
---
|
|
|
|
## Storage Layout on VM 102
|
|
|
|
| Host Path | Purpose |
|
|
|---|---|
|
|
| `/opt/docker/compose/<service>/` | Per-service Docker Compose files |
|
|
| `/opt/docker/conf/<service>/` | Per-service configuration files |
|
|
| `/tank/` | Large persistent data (AI models, images, voice data) |
|
|
| `/tank/aimodels/llm/` | GGUF model files (mounted as `/models` in llama-swap) |
|
|
| `/tank/aimodels/img/comfy/` | Image generation models for ComfyUI |
|
|
| `/tank/chromadb/` | ChromaDB persistent vector data |
|
|
| `/tank/vibevoice/` | VibeVoice Hugging Face cache, voices, state |
|
|
| `/opt/docker/backups/chromadb/` | ChromaDB backup archives |
|
|
|
|
---
|
|
|
|
## Network Configuration
|
|
|
|
- **VLAN**: All VMs on VLAN 50 via `vmbr0`
|
|
- **NIC Type**:
|
|
- Linux VMs: `virtio`
|
|
- Windows VMs: `e1000`
|
|
- **Docker Network**: Services on external network `traefik-net`
|
|
- **Docker Host IP**: `10.250.50.70` (VM 102 / ana-ml2 / PFI-ANA_DOCKER)
|
|
- **Bare Metal Inference Server IP**: `10.250.50.54` (physical server, direct GPU access)
|
|
|
|
---
|
|
|
|
## Scripts Reference
|
|
|
|
### ChromaDB Scripts (VM 102)
|
|
| Script | Purpose |
|
|
|---|---|
|
|
| `setup-chromadb.sh` | Deploy ChromaDB container on VM 102 |
|
|
| `backup-chromadb.sh` | Hot backup with 30-day retention |
|
|
| `health-check-chromadb.sh` | Health monitoring with alerting |
|
|
| `quickstart-chromadb.sh` | One-command ChromaDB setup wrapper |
|
|
|
|
### Utility Scripts
|
|
| Script | Purpose |
|
|
|---|---|
|
|
| `nemotron-query.sh` | Bash client for querying Nemotron via llama-swap API |
|
|
| `inference-hardware-profile.sh` | Comprehensive hardware profiler for AI/ML inference planning |
|
|
|
|
### inference-hardware-profile.sh
|
|
|
|
**Target Hosts:**
|
|
- **Bare metal inferencing server** at `10.250.50.54` — primary target (direct GPU access)
|
|
- **VM 102 (PFI-ANA_DOCKER / ana-ml2)** at `10.250.50.70` — secondary target (GPU passthrough)
|
|
|
|
A comprehensive system hardware profiler designed for AI/ML inference capacity planning. Captures the full hardware picture needed to evaluate and optimize GPU inference workloads.
|
|
|
|
**What it gathers:**
|
|
- System info (OS, kernel, uptime)
|
|
- CPU details (model, cores, threads, clock speed, scaling governor)
|
|
- Memory (total, available, used, swap)
|
|
- Disk storage (all mounts)
|
|
- GPU resources (model, VRAM, utilization, power, temperature, PCIe, clock speeds, throttling, ECC, active processes, MIG mode)
|
|
- Docker container stats (CPU/memory per container)
|
|
- Inference software stack (Python, PyTorch, CUDA, cuDNN, llama.cpp, Ollama, vLLM)
|
|
- Estimated model capacity based on free VRAM (Q4, Q8, FP16)
|
|
- Inference readiness score with actionable recommendations
|
|
|
|
**Output modes:**
|
|
| Flag | Output |
|
|
|---|---|
|
|
| *(default)* | Full colored terminal report |
|
|
| `--json` | Machine-readable JSON |
|
|
| `--quick` | One-line summary |
|
|
| `--gpu-only` | Full report without Docker sections |
|
|
|
|
> **⚠️ Action needed:** Run this script on the **bare metal server** (10.250.50.54) to capture the **physical GPU model, VRAM capacity, and host hardware specs**, which are currently undocumented. The output (especially `--json`) should be saved to close Known Issue #6. Optionally also run on VM 102 (10.250.50.70) for passthrough comparison.
|
|
|
|
---
|
|
|
|
## Known Issues & Observations
|
|
|
|
1. **Nemotron Super** uses `--n-gpu-layers 99` instead of 999 — likely configured to reserve VRAM for other tasks
|
|
2. **Kimi K2** only offloads 2 GPU layers — very CPU-heavy and slow inference
|
|
3. **Granite 4 Small** has `ttl: 0` (persistent caching) but is not assigned to the utility group
|
|
4. **Parakeet STT** references `.env` configuration not tracked in version control
|
|
5. **Qwen3-Coder-Next** download may be incomplete — `.incomplete` files present on disk
|
|
6. **Physical GPU model and VRAM undocumented** — the physical GPU(s) in the bare metal inferencing server at 10.250.50.54 (and those passed through to VM 102 at 10.250.50.70) are not recorded anywhere in project documentation. Run `inference-hardware-profile.sh --json` on the bare metal server and save the output to close this gap.
|
|
|
|
---
|
|
|
|
*This document was generated following a full infrastructure audit. It serves as the primary reference for understanding, maintaining, and extending the PFI network environment.* |