Files

T

Atlas 18265d33ec Update: ATLAS/Infrastructure/Infrastructure-PFI-Project-Index.md — Add Synapse, Element Web, and AIPA Matrix Bridge to VM 102's services list and Docker Services table.

2026-04-11 09:30:28 -07:00

14 KiB

Raw Blame History

created, id, keywords, links, modified, summary, tags, title

created

keywords

links

modified

summary

id: 850d5cfb-1b10-42cb-a3ad-fe4066c8cb17 title: "Infrastructure-PFI — Full Project Index & Architecture Reference" summary: "Comprehensive architecture reference for the Infrastructure-PFI project covering three Tailscale-linked sites, 11 Proxmox VMs at PFI-ANA colo, 9 Docker services with GPU workloads, and llama-swap model management across 20 models." tags: ["infrastructure", "pfi", "docker", "proxmox", "colocation", "reference", "hardware-audit"] keywords: ["docker", "vm", "full", "project", "chromadb", "models", "model", "reference", "compose.yaml", "vcpu", "gb", "gpu", "network", "llama-swap", "bare-metal", "10.250.50.54", "10.250.50.70", "ana-ml2", "PFI-ANA_DOCKER", "inference-server", "VM-102", "RTX-6000-Ada", "EPYC-9254", "hardware-profile"] links: [] created: 2026-04-10T17:35:04+00:00 modified: 2026-04-11T06:15:40+00:00 path: ATLAS/Infrastructure/Infrastructure-PFI-Project-Index.md

Infrastructure-PFI — Full Project Index & Architecture Reference

Overview

Infrastructure-PFI is the configuration and documentation project for the PFI network environment. It covers three physical sites interconnected via Tailscale VPN, hosting a comprehensive infrastructure of Proxmox VMs, Docker services, and AI/ML workloads.

Sites

Site	Network	Hypervisor	Docker Host	Network Gear
ESH-Home	10.0.0.0/16	Proxmox VE	VM at 10.0.50.45	Ubiquiti
PFI-NH3	10.100.0.0/16	Proxmox VE	Proxmox VM	Ubiquiti
PFI-ANA (Colo)	10.250.0.0/16	Proxmox VE at 10.250.250.31:8006	ana-ml2 (bare metal) at 10.250.50.54 · VM 102 (10.250.50.70)	FortiGate

All sites are linked via Tailscale VPN (VM 106 at the colo).

PFI-ANA Colo — Host Reference

Host	IP	Type	Hardware	Role
ana-ml2 (Bare Metal)	10.250.50.54	Physical server	2× EPYC 9254 (48c/96t), 566 GB RAM, 2× RTX 6000 Ada (~90 GB VRAM)	Main inferencing server — direct GPU access
VM 102 (PFI-ANA_DOCKER)	10.250.50.70	Proxmox VM	8 vCPU, 16 GB RAM, 250G disk	Docker host — llama-swap, ComfyUI, VibeVoice, Parakeet, ChromaDB, Dockge, Synapse, Element Web, AIPA Matrix Bridge
VM 100 (PFI-ANA-TRUENAS)	—	Proxmox VM	4 vCPU, 8 GB RAM, 80G disk	NAS/iSCSI storage
VM 101 (PFI-ANA-DC)	—	Proxmox VM	12 vCPU, 24 GB RAM, 240G disk	Windows Domain Controller
VM 104 (PFI-Mongo)	10.250.50.81	Proxmox VM	8 vCPU, 8 GB RAM	MongoDB server
VM 106 (PFI-Tailscale)	—	Proxmox VM	8 vCPU, 2 GB RAM	VPN gateway (Tailscale)

The Proxmox hypervisor management interface is at 10.250.250.31:8006.

Hardware Profile: See Hardware Profile: ana-ml2 for the full bare metal hardware specification (CPU, GPU, memory, software stack, inference capacity estimates).

File & Directory Map

Infrastructure-PFI/
├── README.md                              # Project overview, site summary, structure
├── configs/pfi-ana/
│   ├── proxmox/                           # Raw Proxmox VM configs (11 VMs)
│   │   ├── vm-100.conf  → PFI-ANA-TRUENAS (TrueNAS SCALE, 4 vCPU, 8GB, 80G disk)
│   │   ├── vm-101.conf  → PFI-ANA-DC (Windows 11 DC, 12 vCPU, 24GB, 240G, UEFI+TPM)
│   │   ├── vm-102.conf  → PFI-ANA-Docker (Linux, 8 vCPU, 16GB, 250G — the Docker host)
│   │   ├── vm-103.conf  → PFI-SlaveBot (Windows 10, 8 vCPU, 8GB, 256G)
│   │   ├── vm-104.conf  → PFI-Mongo (Linux, 8 vCPU, 8GB, 256G, MongoDB)
│   │   ├── vm-105.conf  → PFI-Postgres (Debian 11, 16 vCPU, 8GB, 80G, PostgreSQL)
│   │   ├── vm-106.conf  → PFI-Tailscale (Debian 12, 8 vCPU, 2GB, 256G — VPN gateway)
│   │   ├── vm-107.conf  → PFI-Pterodactyl (Debian 11, 8 vCPU, 8GB, 256G — game servers)
│   │   ├── vm-108.conf  → PFI-ANA--DEV (Debian 11, 8 vCPU, 8GB, 120G — dev env)
│   │   ├── vm-110.conf  → PFI-ANA-Webhost (Debian 11, 16 vCPU, 4GB, 250G — web hosting)
│   │   └── vm-111.conf  → pfi-tacticalrmm (Linux, 16 vCPU, 8GB, 256G — RMM)
│   └── docker/
│       ├── compose/                       # Live Docker Compose stacks (managed by Dockge)
│       │   ├── llama-swap/compose.yaml    → LLM gateway (port 9292, CUDA)
│       │   ├── dockge/compose.yaml        → Stack manager (port 5001)
│       │   ├── comfyui/compose.yaml       → Image gen (port 8188, GPU)
│       │   ├── chromadb/compose.yaml      → Vector DB (port 8000, token auth)
│       │   ├── vibevoice/compose.yaml     → TTS (port 8745, GPU)
│       │   └── parakeet/compose.yaml      → STT (port 8300, GPU, local build)
│       ├── conf/
│       │   └── llama-swap/config.yaml     # 20 model definitions, 3 groups
│       └── compose-examples/              # Reference compose files (not live)
│           ├── llama-swap/docker-compose.yml
│           └── chromadb/                  # ChromaDB with Traefik labels + demo app
│               ├── docker-compose.yml
│               ├── docker-compose.demo.yml
│               ├── Dockerfile.demo
│               └── scripts/demo.py
├── docs/pfi-ana/
│   ├── docker-stack.md                    # Docker conventions, 6 services, port map, ASCII diagram
│   ├── proxmox-vms.md                     # 11 VMs, startup order, resource summary, network notes
│   ├── chromadb-setup.md                  # ChromaDB deployment guide, auth, monitoring
│   ├── model-list                         # GGUF inventory with ACTIVE/PENDING/REMOVED status
│   ├── recommended-model-settings.md      # Canonical inference settings for 5 model families
│   ├── vm-102-matrix-synapse.md           # Matrix Synapse deployment on VM 102
│   ├── vm-102-matrix-appservice.md        # Matrix Appservice configuration for AIPA bridge
│   └── hardware-profile-ana-ml2.md        # ✅ Bare metal hardware spec (EPYC, RTX 6000 Ada, VRAM, etc.)
├── scripts/
│   ├── nemotron-query.sh                  # Bash client for Nemotron via llama-swap API
│   ├── setup-chromadb.sh                  # Deploy ChromaDB on VM 102
│   ├── backup-chromadb.sh                 # Hot backup with 30-day retention
│   ├── health-check-chromadb.sh           # Health monitoring with alerting
│   ├── quickstart-chromadb.sh             # One-command ChromaDB setup wrapper
│   └── inference-hardware-profile.sh      # Hardware profiler for AI/ML inference (ana-ml2 bare metal 10.250.50.54)
└── output/                                # Generated output and logs (empty)

Proxmox VM Startup Order

Ordered startup is configured to ensure dependencies are met before dependent services start.

Order	VMID	Name	Delay	Onboot
2	100	TRUENAS	120s	No
3	104	Mongo	60s	Yes
3	105	Postgres	60s	Yes
4	102	Docker	—	Yes
5	101	DC (Windows)	120s	Yes
10	108	DEV	60s	No
20	111	Tactical RMM	—	Yes
30	110	Webhost	120s	Yes

Note: VMs 103 (SlaveBot), 106 (Tailscale), and 107 (Pterodactyl) start via onboot but without ordered delays.

Docker Services (VM 102: PFI-ANA_DOCKER)

Nine services run on the Docker host, with several leveraging GPU acceleration for AI workloads.

Service	Port	GPU	Image
Dockge	5001	No	`louislam/dockge:latest`
llama-swap	9292→8080	Yes (nvidia)	`ghcr.io/mostlygeek/llama-swap:cuda`
ComfyUI	8188	Yes	`mmartial/comfyui-nvidia-docker`
ChromaDB	8000	No	`chromadb/chroma:latest`
VibeVoice	8745	Yes	`eworkerinc/vibevoice:latest`
Parakeet STT	8300→8000	Yes	`parakeet-stt` (local build)
Synapse	8008	No	`matrixdotorg/synapse`
Element Web	8080	No	`vectorim/element-web`
AIPA Matrix Bridge	8009	No	AIPA local

Network Note: All Docker services run on the traefik-net external network.

Matrix on VM 102: See VM 102 — Matrix Synapse Deployment and VM 102 — Matrix Appservice Configuration for full deployment details.

llama-swap Model Inventory

The llama-swap gateway manages 20 GGUF models across multiple categories:

Named Groups

high-reasoning: 4 models (primary reasoning workloads)
heavy-moe: 4 models (Mixture-of-Expert architectures)
utility: 3 persistent models (common utility tasks)

Storage Layout on VM 102

Host Path	Purpose
`/opt/docker/compose/<service>/`	Per-service Docker Compose files
`/opt/docker/conf/<service>/`	Per-service configuration files
`/tank/`	Large persistent data (AI models, images, voice data)
`/tank/aimodels/llm/`	GGUF model files (mounted as `/models` in llama-swap)
`/tank/aimodels/img/comfy/`	Image generation models for ComfyUI
`/tank/chromadb/`	ChromaDB persistent vector data
`/tank/vibevoice/`	VibeVoice Hugging Face cache, voices, state
`/opt/docker/backups/chromadb/`	ChromaDB backup archives

Network Configuration

VLAN: All VMs on VLAN 50 via vmbr0
NIC Type:
- Linux VMs: virtio
- Windows VMs: e1000
Docker Network: Services on external network traefik-net
Docker Host IP: 10.250.50.70 (VM 102 / PFI-ANA_DOCKER)
Bare Metal Inference Server IP: ana-ml2 at 10.250.50.54 (physical server, direct GPU access)

Scripts Reference

ChromaDB Scripts (VM 102)

Script	Purpose
`setup-chromadb.sh`	Deploy ChromaDB container on VM 102
`backup-chromadb.sh`	Hot backup with 30-day retention
`health-check-chromadb.sh`	Health monitoring with alerting
`quickstart-chromadb.sh`	One-command ChromaDB setup wrapper

Utility Scripts

Script	Purpose
`nemotron-query.sh`	Bash client for querying Nemotron via llama-swap API
`inference-hardware-profile.sh`	Comprehensive hardware profiler for AI/ML inference planning

inference-hardware-profile.sh

Target Hosts:

ana-ml2 (bare metal inferencing server) at 10.250.50.54 — primary target (direct GPU access)
VM 102 (PFI-ANA_DOCKER) at 10.250.50.70 — secondary target (GPU passthrough)

A comprehensive system hardware profiler designed for AI/ML inference capacity planning. Captures the full hardware picture needed to evaluate and optimize GPU inference workloads.

What it gathers:

System info (OS, kernel, uptime)
CPU details (model, cores, threads, clock speed, scaling governor)
Memory (total, available, used, swap)
Disk storage (all mounts)
GPU resources (model, VRAM, utilization, power, temperature, PCIe, clock speeds, throttling, ECC, active processes, MIG mode)
Docker container stats (CPU/memory per container)
Inference software stack (Python, PyTorch, CUDA, cuDNN, llama.cpp, Ollama, vLLM)
Estimated model capacity based on free VRAM (Q4, Q8, FP16)
Inference readiness score with actionable recommendations

Output modes:

Flag	Output
(default)	Full colored terminal report
`--json`	Machine-readable JSON
`--quick`	One-line summary
`--gpu-only`	Full report without Docker sections

✅ Completed: Hardware profile captured for ana-ml2 on 2026-04-10. See Hardware Profile: ana-ml2. Optionally run on VM 102 (10.250.50.70) for passthrough comparison.

Known Issues & Observations

Nemotron Super uses --n-gpu-layers 99 instead of 999 — likely configured to reserve VRAM for other tasks
Kimi K2 only offloads 2 GPU layers — very CPU-heavy and slow inference
Granite 4 Small has ttl: 0 (persistent caching) but is not assigned to the utility group
Parakeet STT references .env configuration not tracked in version control
Qwen3-Coder-Next download may be incomplete — .incomplete files present on disk
~~Physical GPU model and VRAM undocumented~~ — ✅ RESOLVED (2026-04-10): Full hardware profile captured via inference-hardware-profile.sh. ana-ml2 has 2× NVIDIA RTX 6000 Ada (~90 GB combined VRAM) on dual EPYC 9254 with 566 GB RAM. See Hardware Profile: ana-ml2. Remaining action: run profile on VM 102 (10.250.50.70) to document passthrough allocation.

This document was generated following a full infrastructure audit. It serves as the primary reference for understanding, maintaining, and extending the PFI network environment.

14 KiB Raw Blame History Unescape Escape