vh/VuongKB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5aecf76d8e611c66821b4e3c539f31c4572e222c
VuongKB/ATLAS/Infrastructure/Infrastructure-PFI-Project-Index.md
T

12 KiB
Raw Blame History

created, id, keywords, links, modified, summary, tags, title
created id keywords links modified summary tags title
2026-04-10T17:35:04+00:00 850d5cfb-1b10-42cb-a3ad-fe4066c8cb17
docker
vm
full
project
chromadb
models
model
reference
compose.yaml
vcpu
gb
gpu
network
llama-swap
bare-metal
10.250.50.54
10.250.50.70
ana-ml2
PFI-ANA_DOCKER
inference-server
VM-102
2026-04-11T01:33:45+00:00 Comprehensive architecture reference for the Infrastructure-PFI project covering three Tailscale-linked sites, 11 Proxmox VMs at PFI-ANA colo, 6 Docker services with GPU workloads, and llama-swap model management across 20 models.
infrastructure
pfi
docker
proxmox
colocation
reference
hardware-audit
Infrastructure-PFI — Full Project Index & Architecture Reference

Infrastructure-PFI — Full Project Index & Architecture Reference

Overview

Infrastructure-PFI is the configuration and documentation project for the PFI network environment. It covers three physical sites interconnected via Tailscale VPN, hosting a comprehensive infrastructure of Proxmox VMs, Docker services, and AI/ML workloads.

Sites

Site Network Hypervisor Docker Host Network Gear
ESH-Home 10.0.0.0/16 Proxmox VE VM at 10.0.50.45 Ubiquiti
PFI-NH3 10.100.0.0/16 Proxmox VE Proxmox VM Ubiquiti
PFI-ANA (Colo) 10.250.0.0/16 Proxmox VE at 10.250.250.31:8006 Bare metal at 10.250.50.54 · VM 102 (10.250.50.70, ana-ml2) FortiGate

All sites are linked via Tailscale VPN (VM 106 at the colo).

PFI-ANA Colo — Host Reference

Host IP Type Role
Bare Metal 10.250.50.54 Physical server Main inferencing server — direct GPU access, runs inference workloads
VM 102 (PFI-ANA_DOCKER) 10.250.50.70 Proxmox VM Docker host — llama-swap, ComfyUI, VibeVoice, Parakeet, ChromaDB, Dockge
VM 100 (PFI-ANA-TRUENAS) — Proxmox VM NAS/iSCSI storage
VM 101 (PFI-ANA-DC) — Proxmox VM Windows Domain Controller
VM 104 (PFI-Mongo) 10.250.50.81 Proxmox VM MongoDB server
VM 106 (PFI-Tailscale) — Proxmox VM VPN gateway (Tailscale)

The Proxmox hypervisor management interface is at 10.250.250.31:8006.

File & Directory Map

Infrastructure-PFI/
├── README.md                              # Project overview, site summary, structure
├── configs/pfi-ana/
│   ├── proxmox/                           # Raw Proxmox VM configs (11 VMs)
│   │   ├── vm-100.conf  → PFI-ANA-TRUENAS (TrueNAS SCALE, 4 vCPU, 8GB, 80G disk)
│   │   ├── vm-101.conf  → PFI-ANA-DC (Windows 11 DC, 12 vCPU, 24GB, 240G, UEFI+TPM)
│   │   ├── vm-102.conf  → PFI-ANA-Docker (Linux, 8 vCPU, 16GB, 250G — the Docker host)
│   │   ├── vm-103.conf  → PFI-SlaveBot (Windows 10, 8 vCPU, 8GB, 256G)
│   │   ├── vm-104.conf  → PFI-Mongo (Linux, 8 vCPU, 8GB, 256G, MongoDB)
│   │   ├── vm-105.conf  → PFI-Postgres (Debian 11, 16 vCPU, 8GB, 80G, PostgreSQL)
│   │   ├── vm-106.conf  → PFI-Tailscale (Debian 12, 8 vCPU, 2GB, 256G — VPN gateway)
│   │   ├── vm-107.conf  → PFI-Pterodactyl (Debian 11, 8 vCPU, 8GB, 256G — game servers)
│   │   ├── vm-108.conf  → PFI-ANA--DEV (Debian 11, 8 vCPU, 8GB, 120G — dev env)
│   │   ├── vm-110.conf  → PFI-ANA-Webhost (Debian 11, 16 vCPU, 4GB, 250G — web hosting)
│   │   └── vm-111.conf  → pfi-tacticalrmm (Linux, 16 vCPU, 8GB, 256G — RMM)
│   └── docker/
│       ├── compose/                       # Live Docker Compose stacks (managed by Dockge)
│       │   ├── llama-swap/compose.yaml    → LLM gateway (port 9292, CUDA)
│       │   ├── dockge/compose.yaml        → Stack manager (port 5001)
│       │   ├── comfyui/compose.yaml       → Image gen (port 8188, GPU)
│       │   ├── chromadb/compose.yaml      → Vector DB (port 8000, token auth)
│       │   ├── vibevoice/compose.yaml     → TTS (port 8745, GPU)
│       │   └── parakeet/compose.yaml      → STT (port 8300, GPU, local build)
│       ├── conf/
│       │   └── llama-swap/config.yaml     # 20 model definitions, 3 groups
│       └── compose-examples/              # Reference compose files (not live)
│           ├── llama-swap/docker-compose.yml
│           └── chromadb/                  # ChromaDB with Traefik labels + demo app
│               ├── docker-compose.yml
│               ├── docker-compose.demo.yml
│               ├── Dockerfile.demo
│               └── scripts/demo.py
├── docs/pfi-ana/
│   ├── docker-stack.md                    # Docker conventions, 6 services, port map, ASCII diagram
│   ├── proxmox-vms.md                     # 11 VMs, startup order, resource summary, network notes
│   ├── chromadb-setup.md                  # ChromaDB deployment guide, auth, monitoring
│   ├── model-list                         # GGUF inventory with ACTIVE/PENDING/REMOVED status
│   └── recommended-model-settings.md      # Canonical inference settings for 5 model families
├── scripts/
│   ├── nemotron-query.sh                  # Bash client for Nemotron via llama-swap API
│   ├── setup-chromadb.sh                  # Deploy ChromaDB on VM 102
│   ├── backup-chromadb.sh                 # Hot backup with 30-day retention
│   ├── health-check-chromadb.sh           # Health monitoring with alerting
│   ├── quickstart-chromadb.sh             # One-command ChromaDB setup wrapper
│   └── inference-hardware-profile.sh      # Hardware profiler for AI/ML inference (bare metal 10.250.50.54)
└── output/                                # Generated output and logs (empty)

Proxmox VM Startup Order

Ordered startup is configured to ensure dependencies are met before dependent services start.

Order VMID Name Delay Onboot
2 100 TRUENAS 120s No
3 104 Mongo 60s Yes
3 105 Postgres 60s Yes
4 102 Docker — Yes
5 101 DC (Windows) 120s Yes
10 108 DEV 60s No
20 111 Tactical RMM — Yes
30 110 Webhost 120s Yes

Note: VMs 103 (SlaveBot), 106 (Tailscale), and 107 (Pterodactyl) start via onboot but without ordered delays.

Docker Services (VM 102: ana-ml2)

Six services run on the Docker host, with most leveraging GPU acceleration for AI workloads.

Service Port GPU Image
Dockge 5001 No louislam/dockge:latest
llama-swap 9292→8080 Yes (nvidia) ghcr.io/mostlygeek/llama-swap:cuda
ComfyUI 8188 Yes mmartial/comfyui-nvidia-docker
ChromaDB 8000 No chromadb/chroma:latest
VibeVoice 8745 Yes eworkerinc/vibevoice:latest
Parakeet STT 8300→8000 Yes parakeet-stt (local build)

Network Note: All Docker services run on the traefik-net external network.

llama-swap Model Inventory

The llama-swap gateway manages 20 GGUF models across multiple categories:

By Category

  • Qwen 3.5: 4 models
  • Nemotron: 2 models
  • Gemma 4: 2 models
  • GLM: 2 models
  • Skyfall: 2 models
  • Coder: 1 model
  • Large/Special: 1 model
  • Granite: 2 models
  • Embeddings: 2 models
  • Reranking: 1 model
  • Pending: 1 (Qwen3-Coder-Next download)

Named Groups

  • high-reasoning: 4 models (primary reasoning workloads)
  • heavy-moe: 4 models (Mixture-of-Expert architectures)
  • utility: 3 persistent models (common utility tasks)

Storage Layout on VM 102

Host Path Purpose
/opt/docker/compose/<service>/ Per-service Docker Compose files
/opt/docker/conf/<service>/ Per-service configuration files
/tank/ Large persistent data (AI models, images, voice data)
/tank/aimodels/llm/ GGUF model files (mounted as /models in llama-swap)
/tank/aimodels/img/comfy/ Image generation models for ComfyUI
/tank/chromadb/ ChromaDB persistent vector data
/tank/vibevoice/ VibeVoice Hugging Face cache, voices, state
/opt/docker/backups/chromadb/ ChromaDB backup archives

Network Configuration

  • VLAN: All VMs on VLAN 50 via vmbr0
  • NIC Type:
    • Linux VMs: virtio
    • Windows VMs: e1000
  • Docker Network: Services on external network traefik-net
  • Docker Host IP: 10.250.50.70 (VM 102 / ana-ml2 / PFI-ANA_DOCKER)
  • Bare Metal Inference Server IP: 10.250.50.54 (physical server, direct GPU access)

Scripts Reference

ChromaDB Scripts (VM 102)

Script Purpose
setup-chromadb.sh Deploy ChromaDB container on VM 102
backup-chromadb.sh Hot backup with 30-day retention
health-check-chromadb.sh Health monitoring with alerting
quickstart-chromadb.sh One-command ChromaDB setup wrapper

Utility Scripts

Script Purpose
nemotron-query.sh Bash client for querying Nemotron via llama-swap API
inference-hardware-profile.sh Comprehensive hardware profiler for AI/ML inference planning

inference-hardware-profile.sh

Target Hosts:

  • Bare metal inferencing server at 10.250.50.54 — primary target (direct GPU access)
  • VM 102 (PFI-ANA_DOCKER / ana-ml2) at 10.250.50.70 — secondary target (GPU passthrough)

A comprehensive system hardware profiler designed for AI/ML inference capacity planning. Captures the full hardware picture needed to evaluate and optimize GPU inference workloads.

What it gathers:

  • System info (OS, kernel, uptime)
  • CPU details (model, cores, threads, clock speed, scaling governor)
  • Memory (total, available, used, swap)
  • Disk storage (all mounts)
  • GPU resources (model, VRAM, utilization, power, temperature, PCIe, clock speeds, throttling, ECC, active processes, MIG mode)
  • Docker container stats (CPU/memory per container)
  • Inference software stack (Python, PyTorch, CUDA, cuDNN, llama.cpp, Ollama, vLLM)
  • Estimated model capacity based on free VRAM (Q4, Q8, FP16)
  • Inference readiness score with actionable recommendations

Output modes:

Flag Output
(default) Full colored terminal report
--json Machine-readable JSON
--quick One-line summary
--gpu-only Full report without Docker sections

⚠️ Action needed: Run this script on the bare metal server (10.250.50.54) to capture the physical GPU model, VRAM capacity, and host hardware specs, which are currently undocumented. The output (especially --json) should be saved to close Known Issue #6. Optionally also run on VM 102 (10.250.50.70) for passthrough comparison.

Known Issues & Observations

  1. Nemotron Super uses --n-gpu-layers 99 instead of 999 — likely configured to reserve VRAM for other tasks
  2. Kimi K2 only offloads 2 GPU layers — very CPU-heavy and slow inference
  3. Granite 4 Small has ttl: 0 (persistent caching) but is not assigned to the utility group
  4. Parakeet STT references .env configuration not tracked in version control
  5. Qwen3-Coder-Next download may be incomplete — .incomplete files present on disk
  6. Physical GPU model and VRAM undocumented — the physical GPU(s) in the bare metal inferencing server at 10.250.50.54 (and those passed through to VM 102 at 10.250.50.70) are not recorded anywhere in project documentation. Run inference-hardware-profile.sh --json on the bare metal server and save the output to close this gap.

This document was generated following a full infrastructure audit. It serves as the primary reference for understanding, maintaining, and extending the PFI network environment.

Reference in New Issue View Git Blame Copy Permalink