vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
esh-pfi-infrastructure/docs/pfi/docker-stack.md
T
vh 91bda3c480 fv-ml1: complete the cutover — rename, renumber, DNS, and the LiteLLM repoint 
The box is physically at Fountain Valley, renamed, renumbered onto 10.251/16,
and serving inference again. This lands the repo half of that.

Host: hostname ana-ml2 -> fv-ml1, pinned to 10.251.50.54 by a dnsmasq
reservation so the address the runbook, DNS and LiteLLM all assume is the
address it actually has. Its headscale node is renamed too.

The sweep ran from scripts/fv-ml1-rename-sweep.sh, whose allowlist is the
reason this diff touches current-state files and not the record. Dated
persistent-memory entries, archival-memory and incident notes still say
ana-ml2 in 31 and 62 places respectively, because that is what the box was
when those things happened. Rewriting them would make the history lie.

LiteLLM was the load-bearing piece and needed more than the api_base sed the
runbook describes. Twenty api_base entries repointed, but a grep-and-verify
pass also caught a LIVE pass_through_endpoints target for the scalar-judge
reward route still on the old address -- an api_base-only substitution would
have left it dead. Four prose references describing current state were
repointed as well; one historical note recording where a hand-test was run
is deliberately left pointing at 10.250.50.54.

Two facts in the server tables were wrong and are corrected here. The site is
Fountain Valley, not Anaheim. And the box has FOUR RTX PRO 6000 Blackwell
Max-Q, not two -- verified by nvidia-smi -L and independently by PCI
enumeration of four GB202GL devices. That is 391 GB of VRAM rather than 196,
which changes what fits on it.

DNS: fv-ml1, fv-ml1-bmc and fv-gw added under the fv site via the piggyback
approach, scriberr re-homed, and the ana-ml2 records removed. Applied to all
three resolvers. The BMC record carries a warning that its 802.1q VLAN tag
must stay disabled -- it shipped tagging VLAN 250 into an untagged port,
which made it invisible to every network-side diagnostic and is the reason
it appeared dead through several cable changes.

Verified end to end: summarizer and sec both answer through the Anaheim
gateway across the mesh to FV seats on different ports.
2026-09-12 22:00:50 -07:00

314 lines
12 KiB
Markdown
Raw Permalink Blame History

# PFI-ANA Docker Stack
## Overview
PFI-ANA (the Colo) runs Docker services managed through **Dockge**, a compose-aware Docker management UI. All services that require inbound HTTP/HTTPS routing join a shared external Docker network called `traefik-net`, allowing **Traefik** to act as a reverse proxy and handle TLS termination and routing.
## Conventions
### Network
| Network | Docker Name | Purpose |
|---|---|---|
| Traefik network | `traefik-net` | Shared external network. Services join as `tnet` so Traefik can discover them. |
Every compose file that needs to be reachable through Traefik **must** include:
```yaml
networks:
tnet:
name: traefik-net
external: true
```
And the service must list `tnet` under its `networks` key.
### Storage Paths (Host)
| Host Path | Purpose |
|---|---|
| `/opt/docker/compose/<service>/` | Per-service compose files (managed by Dockge) |
| `/opt/docker/conf/<service>/` | Per-service configuration files (bind-mounted into containers) |
| `/tank/` | Large / persistent data storage (e.g., AI models, generated images, voice data) |
### GPU Support
Services requiring GPU access use the **NVIDIA Container Toolkit**:
```yaml
runtime: nvidia
```
or the more explicit device reservation:
```yaml
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
```
### Homepage Dashboard Labels
Several services include Docker labels for **Homepage** (a dashboard UI). The convention is:
```yaml
labels:
- homepage.group=<Group Name>
- homepage.name=<Display Name>
- homepage.icon=<icon identifier>
- homepage.description=<brief description>
- homepage.href=http://<host>:<port>
```
All services reference the VM 102 host IP `10.250.50.70` (PFI-ANA_DOCKER).
### Compose File Location
Dockge expects compose files under `/opt/docker/compose/` on the Docker host. The live compose files are version-controlled in this project under `configs/pfi-ana/docker/compose/`. Configuration files that containers bind-mount live under `configs/pfi-ana/docker/conf/`.
---
## Live Services Index
| Service | Host Port | Container Port | GPU | Homepage Group | Status | Compose File | Config File |
|---|---|---|---|---|---|---|---|
| **Dockge** | 5001 | 5001 | No | PFI-ANA | ✅ Live | `compose/dockge/compose.yaml` | — |
| **llama-swap** | 9292 | 8080 | Yes (CUDA) | — | ✅ Live | `compose/llama-swap/compose.yaml` | `conf/llama-swap/config.yaml` |
| **ComfyUI** | 8188 | 8188 | Yes (all caps) | AI Systems | ✅ Live | `compose/comfyui/compose.yaml` | — |
| **VibeVoice** | 8745 | 8745 | Yes (gpu) | AI Systems | ✅ Live | `compose/vibevoice/compose.yaml` | — |
| **Parakeet STT** | 8300 | 8000 | Yes (gpu) | AI Systems | ✅ Live | `compose/parakeet/compose.yaml` | — |
| **ChromaDB** | 8000 | 8000 | No | AI Systems | ✅ Live | `compose/chromadb/compose.yaml` | `/opt/docker/conf/chromadb/auth_token` |
> All paths relative to `configs/pfi-ana/docker/`.
---
## Service Details
### 1. Dockge — Docker Compose Management UI
- **Image**: `louislam/dockge:latest`
- **Port**: 5001 → 5001
- **Restart policy**: `unless-stopped`
- **Homepage group**: PFI-ANA
- **Compose file**: `compose/dockge/compose.yaml`
**Volumes**:
| Host / Volume | Container | Purpose |
|---|---|---|
| `/var/run/docker.sock` | `/var/run/docker.sock` | Docker socket for managing containers |
| `dockge_data` (named volume) | `/app/data` | Dockge application data |
| `/opt/docker/compose` | `/opt/docker/compose` | Compose stack directory |
**Environment**:
- `DOCKGE_STACKS_DIR=/opt/docker/compose` — tells Dockge where to find/manage compose stacks
**Notes**: Dockge is the management interface for all other compose stacks on this host. It has full Docker daemon access via the socket mount.
---
### 2. llama-swap — Multi-Model LLM Gateway
- **Image**: `ghcr.io/mostlygeek/llama-swap:cuda`
- **Port**: 9292 → 8080
- **Runtime**: `nvidia` (CUDA)
- **Compose file**: `compose/llama-swap/compose.yaml`
- **Config file**: `conf/llama-swap/config.yaml`
- **Interactive**: `stdin_open: true`, `tty: true` (required by llama-swap)
**Volumes**:
| Host Path | Container Path | Purpose |
|---|---|---|
| `/opt/docker/conf/llama-swap/config.yaml` | `/app/config.yaml` | llama-swap configuration (models, groups, params) |
| `/tank/aimodels/llm` | `/models` | LLM model files (GGUF format) |
**Configured Models:** the live list (with quants, context, TTLs,
group memberships) is in
[`stacks/llama-swap/conf/config.yaml`](../../stacks/llama-swap/conf/config.yaml).
Don't mirror the table here — it drifted last time and immediately
went stale. Live HTTP listing:
`curl http://10.251.50.54:9292/v1/models`.
**Model Groups**:
| Group | Swap | Exclusive | Persistent | Members |
|---|---|---|---|---|
| `high-reasoning` | false | false | — | qwen3.5-35-a3b, qwen3.5-35-a3b-code, gemma4-31b-dense |
| `utility` | false | false | ✅ | embeddinggemma-300M, bge-reranker-v2-m3-0.6B |
**Global Settings**:
- `healthCheckTimeout`: 1200 seconds (20 minutes) — long timeout for large models
- `logLevel`: info
- `metricsMaxInMemory`: 1000
---
### 3. ComfyUI — Image Generation UI
- **Image**: `mmartial/comfyui-nvidia-docker:ubuntu24_cuda13.0-latest`
- **Port**: 8188 → 8188
- **Runtime**: `nvidia` with full device reservation (gpu, compute, utility capabilities)
- **Restart policy**: `unless-stopped`
- **Homepage group**: AI Systems
- **Compose file**: `compose/comfyui/compose.yaml`
**Volumes**:
| Host Path | Container Path | Purpose |
|---|---|---|
| `/tank/comfy/run` | `/comfy/mnt` | ComfyUI workspace / output directory |
| `/tank/aimodels/img/comfy` | `/basedir` | Image models and ComfyUI base directory |
**Environment**:
| Variable | Value | Purpose |
|---|---|---|
| `WANTED_UID` | 1001 | Run as user ID 1001 |
| `WANTED_GID` | 1002 | Run as group ID 1002 |
| `BASE_DIRECTORY` | /basedir | ComfyUI base directory path |
| `SECURITY_LEVEL` | weak | Relaxed security (private network) |
| `NVIDIA_VISIBLE_DEVICES` | all | Expose all GPUs |
| `NVIDIA_DRIVER_CAPABILITIES` | all | Enable all GPU capabilities |
**Notes**: Runs with user-mapped permissions (UID 1001 / GID 1002). The `basedir` points to the image model storage on `/tank`.
---
### 4. VibeVoice — Voice/Audio AI Service
- **Image**: `eworkerinc/vibevoice:latest`
- **Container name**: `vibevoice`
- **Port**: 8745 → 8745
- **GPU**: Yes (all devices, gpu capability)
- **Restart policy**: `unless-stopped`
- **Homepage group**: AI Systems
- **Compose file**: `compose/vibevoice/compose.yaml`
**Volumes**:
| Host Path | Container Path | Purpose |
|---|---|---|
| `/tank/vibevoice/hf` | `/root/.cache/huggingface` | HuggingFace model cache |
| `/tank/vibevoice/voices` | `/app/voices` | Voice data / presets |
| `/tank/vibevoice/state` | `/var/lib/eworker` | Application state persistence |
**Environment**:
| Variable | Value | Purpose |
|---|---|---|
| `ENABLE_1_5B` | true | Enable 1.5B parameter voice model |
| `ENABLE_LARGE` | true | Enable large voice model |
| `AUTH_REQUIRED` | true | Require authentication |
| `CORS_ENABLED` | true | Enable CORS headers |
| `ALLOWED_ORIGINS` | * | Allow all origins (development/private network) |
---
### 5. Parakeet STT — Speech-to-Text Service
- **Image**: `parakeet-stt` (locally built)
- **Port**: 8300 → 8000
- **GPU**: Yes (all devices, gpu capability)
- **Restart policy**: `unless-stopped`
- **Homepage group**: AI Systems
- **Compose file**: `compose/parakeet/compose.yaml`
- **Env file**: `.env` (not tracked in project — likely contains API keys or model config)
**Volumes**:
| Volume | Container Path | Purpose |
|---|---|---|
| `parakeet_cache` (named volume) | `/root/.cache` | Model download cache |
**Notes**: Uses a locally-built image (no registry prefix). The `.env` file is referenced but not stored in the project — it likely contains environment-specific configuration on the Docker host.
---
### 6. ChromaDB — Vector Database
- **Image**: `chromadb/chroma:latest`
- **Container name**: `chromadb`
- **Port**: 8000 → 8000
- **GPU**: No
- **Restart policy**: `unless-stopped`
- **Homepage group**: AI Systems
- **Compose file**: `compose/chromadb/compose.yaml`
**Volumes**:
| Host Path | Container Path | Mode | Purpose |
|---|---|---|---|
| `/opt/docker/conf/chromadb` | `/conf` | read-only | Config directory (contains auth_token) |
| `/tank/chromadb` | `/data` | read-write | Persistent vector data |
**Environment**:
| Variable | Value | Purpose |
|---|---|---|
| `CHROMA_SERVER_AUTHN_CREDENTIALS_FILE` | `/conf/auth_token` | Path to auth token file inside container |
| `CHROMA_SERVER_AUTHN_PROVIDER` | `chromadb.server.auth.token.TokenAuthenticationServerProvider` | Enable token-based authentication |
| `IS_PERSISTENT` | TRUE | Enable persistent storage |
| `PERSIST_DIRECTORY` | `/data` | Where vector data is stored inside container |
| `ANONYMIZED_TELEMETRY` | FALSE | Disable telemetry |
**Health Check**:
| Setting | Value |
|---|---|
| Test | `curl -f http://localhost:8000/api/v1/health` |
| Interval | 30s |
| Timeout | 10s |
| Retries | 3 |
| Start period | 40s |
**Authentication**: Token-based. The auth token is stored at `/opt/docker/conf/chromadb/auth_token` on the host (mode 600), generated with `openssl rand -hex 32`. Clients must supply this token to access the API.
**Notes**: CPU-only service (no GPU). Token is generated once at deploy with `openssl rand -hex 32` into `/opt/docker/conf/chromadb/auth_token` (mode 600); clients pass it via the chromadb client `Settings(chroma_client_auth_provider="...TokenAuthClientProvider", chroma_client_auth_credentials="...")`.
---
## Infrastructure Summary
```
┌──────────────────────────────────────────────────────────────┐
│ PFI-ANA (10.250.50.70) │
│ Docker Host (PFI-ANA_DOCKER) │
│ │
│ ┌──────────┐ Manages all compose stacks │
│ │ Dockge │◄─── /opt/docker/compose/* │
│ │ :5001 │ /var/run/docker.sock │
│ └──────────┘ │
│ │
│ ┌──────────────────┐ GPU via passthrough │
│ │ llama-swap :9292 (CUDA, multi-model gateway) │ │
│ │ ├── 15 models (chat, code, embedding, reranker) │ │
│ │ ├── 2 groups (high-reasoning, utility) │ │
│ │ └── Models from /tank/aimodels/llm │ │
│ └──────────────────┘ GPU via passthrough │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ GPU via │
│ │ ComfyUI │ │ VibeVoice │ │ Parakeet │ passthrough │
│ │ :8188 │ │ :8745 │ │ :8300 │ │
│ │ (GPU, img) │ │ (GPU, voice)│ │ (GPU, STT) │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │
│ ┌──────────┐ │
│ │ ChromaDB │ Token auth, persistent vectors │
│ │ :8000 │ /tank/chromadb (data) │
│ │ (CPU only) │ /opt/docker/conf/chromadb (config) │
│ └──────────┘ │
│ │
│ ── All services on traefik-net (external) ── │
│ ── /tank/* = persistent large data storage ── │
│ ── /opt/docker/* = config + compose files ── │
└──────────────────────────────────────────────────────────────┘
```
## Port Allocation
| Port | Service | Protocol |
|---|---|---|
| 5001 | Dockge | HTTP |
| 8000 | ChromaDB | HTTP |
| 8188 | ComfyUI | HTTP |
| 8300 | Parakeet STT | HTTP (→ container 8000) |
| 8745 | VibeVoice | HTTP |
| 9292 | llama-swap | HTTP (→ container 8080) |
Reference in New Issue View Git Blame Copy Permalink