vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
dfdabd1d02e35ec8bd85f01d4694f83cb3515d29
esh-pfi-infrastructure/docs/pfi/docker-stack.md
T
vh e376d0aec9 Initial commit: PFI fleet inventory, stacks, tooling, and backup pipeline 
Captures the full workspace state built up to this point:

  - CLAUDE.md + README.md describing conventions and the four-host fleet
    (ana-ml2, ana-docker, nh3-docker, esh-docker-vm).
  - Per-host notes under servers/<host>/ with ssh-target fallback files
    and latest system-details snapshots (two in-compose credential leaks
    scrubbed; the upstream compose files still need to move those to .env).
  - scripts/: server_inspect.sh (read-only remote diagnostic),
    refresh-server-info.sh (dir-driven discovery + snapshot capture with
    validation warnings), add-host.sh, sync-stacks.sh (pull
    compose/conf trees), deploy-stack.sh (push with per-file diff + prompt).
  - stacks/: canonical compose for backrest, beszel, dozzle, llama-swap,
    rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired
    infinity reference. All use the .env-driven + traefik-net + homepage
    label pattern.
  - configs/restic/ana-docker/: first resticprofile config + pre-backup
    hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates
    for the other three hosts to come.
  - docs/pfi/: general infrastructure reference carried over.
  - .gitignore excludes .env, stacks-mirror/, and assorted secret/state
    filenames to prevent re-leaks on later commits.
2026-04-20 14:29:48 -07:00

334 lines
14 KiB
Markdown
Raw Blame History

# PFI-ANA Docker Stack
## Overview
PFI-ANA (the Colo) runs Docker services managed through **Dockge**, a compose-aware Docker management UI. All services that require inbound HTTP/HTTPS routing join a shared external Docker network called `traefik-net`, allowing **Traefik** to act as a reverse proxy and handle TLS termination and routing.
## Conventions
### Network
| Network | Docker Name | Purpose |
|---|---|---|
| Traefik network | `traefik-net` | Shared external network. Services join as `tnet` so Traefik can discover them. |
Every compose file that needs to be reachable through Traefik **must** include:
```yaml
networks:
tnet:
name: traefik-net
external: true
```
And the service must list `tnet` under its `networks` key.
### Storage Paths (Host)
| Host Path | Purpose |
|---|---|
| `/opt/docker/compose/<service>/` | Per-service compose files (managed by Dockge) |
| `/opt/docker/conf/<service>/` | Per-service configuration files (bind-mounted into containers) |
| `/tank/` | Large / persistent data storage (e.g., AI models, generated images, voice data) |
### GPU Support
Services requiring GPU access use the **NVIDIA Container Toolkit**:
```yaml
runtime: nvidia
```
or the more explicit device reservation:
```yaml
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
```
### Homepage Dashboard Labels
Several services include Docker labels for **Homepage** (a dashboard UI). The convention is:
```yaml
labels:
- homepage.group=<Group Name>
- homepage.name=<Display Name>
- homepage.icon=<icon identifier>
- homepage.description=<brief description>
- homepage.href=http://<host>:<port>
```
All services reference the VM 102 host IP `10.250.50.70` (PFI-ANA_DOCKER).
### Compose File Location
Dockge expects compose files under `/opt/docker/compose/` on the Docker host. The live compose files are version-controlled in this project under `configs/pfi-ana/docker/compose/`. Configuration files that containers bind-mount live under `configs/pfi-ana/docker/conf/`.
---
## Live Services Index
| Service | Host Port | Container Port | GPU | Homepage Group | Status | Compose File | Config File |
|---|---|---|---|---|---|---|---|
| **Dockge** | 5001 | 5001 | No | PFI-ANA | ✅ Live | `compose/dockge/compose.yaml` | — |
| **llama-swap** | 9292 | 8080 | Yes (CUDA) | — | ✅ Live | `compose/llama-swap/compose.yaml` | `conf/llama-swap/config.yaml` |
| **ComfyUI** | 8188 | 8188 | Yes (all caps) | AI Systems | ✅ Live | `compose/comfyui/compose.yaml` | — |
| **VibeVoice** | 8745 | 8745 | Yes (gpu) | AI Systems | ✅ Live | `compose/vibevoice/compose.yaml` | — |
| **Parakeet STT** | 8300 | 8000 | Yes (gpu) | AI Systems | ✅ Live | `compose/parakeet/compose.yaml` | — |
| **ChromaDB** | 8000 | 8000 | No | AI Systems | ✅ Live | `compose/chromadb/compose.yaml` | `/opt/docker/conf/chromadb/auth_token` |
> All paths relative to `configs/pfi-ana/docker/`.
---
## Service Details
### 1. Dockge — Docker Compose Management UI
- **Image**: `louislam/dockge:latest`
- **Port**: 5001 → 5001
- **Restart policy**: `unless-stopped`
- **Homepage group**: PFI-ANA
- **Compose file**: `compose/dockge/compose.yaml`
**Volumes**:
| Host / Volume | Container | Purpose |
|---|---|---|
| `/var/run/docker.sock` | `/var/run/docker.sock` | Docker socket for managing containers |
| `dockge_data` (named volume) | `/app/data` | Dockge application data |
| `/opt/docker/compose` | `/opt/docker/compose` | Compose stack directory |
**Environment**:
- `DOCKGE_STACKS_DIR=/opt/docker/compose` — tells Dockge where to find/manage compose stacks
**Notes**: Dockge is the management interface for all other compose stacks on this host. It has full Docker daemon access via the socket mount.
---
### 2. llama-swap — Multi-Model LLM Gateway
- **Image**: `ghcr.io/mostlygeek/llama-swap:cuda`
- **Port**: 9292 → 8080
- **Runtime**: `nvidia` (CUDA)
- **Compose file**: `compose/llama-swap/compose.yaml`
- **Config file**: `conf/llama-swap/config.yaml`
- **Interactive**: `stdin_open: true`, `tty: true` (required by llama-swap)
**Volumes**:
| Host Path | Container Path | Purpose |
|---|---|---|
| `/opt/docker/conf/llama-swap/config.yaml` | `/app/config.yaml` | llama-swap configuration (models, groups, params) |
| `/tank/aimodels/llm` | `/models` | LLM model files (GGUF format) |
**Configured Models** (from `config.yaml`):
| Model ID | Display Name | Quantization | Context Size | TTL (s) | Notes |
|---|---|---|---|---|---|
| `qwen3-4b` | Qwen3-4B-Instruct-2507-Q6_K | Q6_K | default | 0 (persistent) | Small general-purpose model |
| `glm4.5-air` | GLM-4.5-Air Q4_K_M | Q4_K_M | 40,000 | 600 | Flash attention enabled |
| `skyfall-r1-31b-q6k` | Skyfall 31B v4 | Q6_K_L | 40,000 | 600 | Flash attention, full GPU offload |
| `GLM-Steam-106B-QK4M-A12B` | GLM-Steam 106B A12B | Q4_K_M | 40,000 | 600 | 2-shard model, MoE with 12B active |
| `kimik2-q2kxl` | Kimi K2 Instruct | UD-Q2_K_XL | default | 600 | 8-shard model, only 2 GPU layers (CPU-heavy) |
| `qwen3-coder-30b-iq4-nl` | Qwen3 Coder 30B A3B | IQ4_NL | 40,000 | 0 (persistent) | MoE 3B active, coding-optimized |
| `unsloth-granite-4-small` | Granite 4.0 Small | Q4_K_M | 120,000 | 0 (persistent) | IBM Granite, deterministic (temp=0) |
| `qwen3.5-35-a3b` | Qwen 3.5 35B A3B | UD-Q4_K_XL | 32,768 | 0 (persistent) | MoE, thinking mode, temp=1.0 |
| `qwen3.5-35-a3b-code` | Qwen 3.5 35B A3B Code | UD-Q4_K_XL | 32,768 | 0 (persistent) | Same model, code-tuned params (temp=0.6) |
| `gemma4-26b-a4b` | Gemma 4 26B A4B | UD-Q4_K_XL | 32,768 | 600 | MoE 4B active, thinking enabled, supports images |
| `gemma4-31b-dense` | Gemma 4 31B Dense | UD-Q4_K_XL | 32,768 | 600 | Full dense model, thinking enabled, supports images |
| `embeddinggemma-300M` | Embedding Gemma 300M | Q8_0 | 2,048 | 0 (persistent) | Embedding model, cls pooling |
| `qwen3-embedding-0.6B` | Qwen3 Embedding 0.6B | Q8_0 | 32,768 | 0 (persistent) | Embedding model, mean pooling |
| `jina-reranker-v3-0.6B` | Jina Reranker v3 | Q8_0 | 32,768 | 0 (persistent) | Reranking model |
| `bge-reranker-v2-m3-0.6B` | BGE Reranker v2 m3 | Q8_0 | 32,768 | 0 (persistent) | Reranking model |
**Model Groups**:
| Group | Swap | Exclusive | Persistent | Members |
|---|---|---|---|---|
| `high-reasoning` | false | false | — | qwen3.5-35-a3b, qwen3.5-35-a3b-code, gemma4-31b-dense |
| `utility` | false | false | ✅ | embeddinggemma-300M, bge-reranker-v2-m3-0.6B |
**Global Settings**:
- `healthCheckTimeout`: 1200 seconds (20 minutes) — long timeout for large models
- `logLevel`: info
- `metricsMaxInMemory`: 1000
---
### 3. ComfyUI — Image Generation UI
- **Image**: `mmartial/comfyui-nvidia-docker:ubuntu24_cuda13.0-latest`
- **Port**: 8188 → 8188
- **Runtime**: `nvidia` with full device reservation (gpu, compute, utility capabilities)
- **Restart policy**: `unless-stopped`
- **Homepage group**: AI Systems
- **Compose file**: `compose/comfyui/compose.yaml`
**Volumes**:
| Host Path | Container Path | Purpose |
|---|---|---|
| `/tank/comfy/run` | `/comfy/mnt` | ComfyUI workspace / output directory |
| `/tank/aimodels/img/comfy` | `/basedir` | Image models and ComfyUI base directory |
**Environment**:
| Variable | Value | Purpose |
|---|---|---|
| `WANTED_UID` | 1001 | Run as user ID 1001 |
| `WANTED_GID` | 1002 | Run as group ID 1002 |
| `BASE_DIRECTORY` | /basedir | ComfyUI base directory path |
| `SECURITY_LEVEL` | weak | Relaxed security (private network) |
| `NVIDIA_VISIBLE_DEVICES` | all | Expose all GPUs |
| `NVIDIA_DRIVER_CAPABILITIES` | all | Enable all GPU capabilities |
**Notes**: Runs with user-mapped permissions (UID 1001 / GID 1002). The `basedir` points to the image model storage on `/tank`.
---
### 4. VibeVoice — Voice/Audio AI Service
- **Image**: `eworkerinc/vibevoice:latest`
- **Container name**: `vibevoice`
- **Port**: 8745 → 8745
- **GPU**: Yes (all devices, gpu capability)
- **Restart policy**: `unless-stopped`
- **Homepage group**: AI Systems
- **Compose file**: `compose/vibevoice/compose.yaml`
**Volumes**:
| Host Path | Container Path | Purpose |
|---|---|---|
| `/tank/vibevoice/hf` | `/root/.cache/huggingface` | HuggingFace model cache |
| `/tank/vibevoice/voices` | `/app/voices` | Voice data / presets |
| `/tank/vibevoice/state` | `/var/lib/eworker` | Application state persistence |
**Environment**:
| Variable | Value | Purpose |
|---|---|---|
| `ENABLE_1_5B` | true | Enable 1.5B parameter voice model |
| `ENABLE_LARGE` | true | Enable large voice model |
| `AUTH_REQUIRED` | true | Require authentication |
| `CORS_ENABLED` | true | Enable CORS headers |
| `ALLOWED_ORIGINS` | * | Allow all origins (development/private network) |
---
### 5. Parakeet STT — Speech-to-Text Service
- **Image**: `parakeet-stt` (locally built)
- **Port**: 8300 → 8000
- **GPU**: Yes (all devices, gpu capability)
- **Restart policy**: `unless-stopped`
- **Homepage group**: AI Systems
- **Compose file**: `compose/parakeet/compose.yaml`
- **Env file**: `.env` (not tracked in project — likely contains API keys or model config)
**Volumes**:
| Volume | Container Path | Purpose |
|---|---|---|
| `parakeet_cache` (named volume) | `/root/.cache` | Model download cache |
**Notes**: Uses a locally-built image (no registry prefix). The `.env` file is referenced but not stored in the project — it likely contains environment-specific configuration on the Docker host.
---
### 6. ChromaDB — Vector Database
- **Image**: `chromadb/chroma:latest`
- **Container name**: `chromadb`
- **Port**: 8000 → 8000
- **GPU**: No
- **Restart policy**: `unless-stopped`
- **Homepage group**: AI Systems
- **Compose file**: `compose/chromadb/compose.yaml`
**Volumes**:
| Host Path | Container Path | Mode | Purpose |
|---|---|---|---|
| `/opt/docker/conf/chromadb` | `/conf` | read-only | Config directory (contains auth_token) |
| `/tank/chromadb` | `/data` | read-write | Persistent vector data |
**Environment**:
| Variable | Value | Purpose |
|---|---|---|
| `CHROMA_SERVER_AUTHN_CREDENTIALS_FILE` | `/conf/auth_token` | Path to auth token file inside container |
| `CHROMA_SERVER_AUTHN_PROVIDER` | `chromadb.server.auth.token.TokenAuthenticationServerProvider` | Enable token-based authentication |
| `IS_PERSISTENT` | TRUE | Enable persistent storage |
| `PERSIST_DIRECTORY` | `/data` | Where vector data is stored inside container |
| `ANONYMIZED_TELEMETRY` | FALSE | Disable telemetry |
**Health Check**:
| Setting | Value |
|---|---|
| Test | `curl -f http://localhost:8000/api/v1/health` |
| Interval | 30s |
| Timeout | 10s |
| Retries | 3 |
| Start period | 40s |
**Authentication**: Token-based. The auth token is stored at `/opt/docker/conf/chromadb/auth_token` on the host (mode 600), generated with `openssl rand -hex 32`. Clients must supply this token to access the API.
**Notes**: CPU-only service (no GPU). The compose-examples directory contains a reference compose file with Traefik labels and a demo app — see [chromadb-setup.md](chromadb-setup.md) for full deployment instructions and the demo.
---
## Infrastructure Summary
```
┌──────────────────────────────────────────────────────────────┐
│ PFI-ANA (10.250.50.70) │
│ Docker Host (PFI-ANA_DOCKER) │
│ │
│ ┌──────────┐ Manages all compose stacks │
│ │ Dockge │◄─── /opt/docker/compose/* │
│ │ :5001 │ /var/run/docker.sock │
│ └──────────┘ │
│ │
│ ┌──────────────────┐ GPU via passthrough │
│ │ llama-swap :9292 (CUDA, multi-model gateway) │ │
│ │ ├── 15 models (chat, code, embedding, reranker) │ │
│ │ ├── 2 groups (high-reasoning, utility) │ │
│ │ └── Models from /tank/aimodels/llm │ │
│ └──────────────────┘ GPU via passthrough │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ GPU via │
│ │ ComfyUI │ │ VibeVoice │ │ Parakeet │ passthrough │
│ │ :8188 │ │ :8745 │ │ :8300 │ │
│ │ (GPU, img) │ │ (GPU, voice)│ │ (GPU, STT) │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │
│ ┌──────────┐ │
│ │ ChromaDB │ Token auth, persistent vectors │
│ │ :8000 │ /tank/chromadb (data) │
│ │ (CPU only) │ /opt/docker/conf/chromadb (config) │
│ └──────────┘ │
│ │
│ ── All services on traefik-net (external) ── │
│ ── /tank/* = persistent large data storage ── │
│ ── /opt/docker/* = config + compose files ── │
└──────────────────────────────────────────────────────────────┘
```
## Port Allocation
| Port | Service | Protocol |
|---|---|---|
| 5001 | Dockge | HTTP |
| 8000 | ChromaDB | HTTP |
| 8188 | ComfyUI | HTTP |
| 8300 | Parakeet STT | HTTP (→ container 8000) |
| 8745 | VibeVoice | HTTP |
| 9292 | llama-swap | HTTP (→ container 8080) |
## Example Configurations
Reference/example compose files (not live) are stored in `configs/pfi-ana/docker/compose-examples/`:
- `compose-examples/llama-swap/docker-compose.yml` — earlier llama-swap reference
- `compose-examples/chromadb/` — ChromaDB with Traefik labels, auth, health checks, and demo app
Full ChromaDB setup instructions: [chromadb-setup.md](chromadb-setup.md)
Reference in New Issue View Git Blame Copy Permalink