esh-pfi-infrastructure/docs/pfi/docker-stack.md

PFI-ANA Docker Stack

Overview

PFI-ANA (the Colo) runs Docker services managed through Dockge, a compose-aware Docker management UI. All services that require inbound HTTP/HTTPS routing join a shared external Docker network called traefik-net, allowing Traefik to act as a reverse proxy and handle TLS termination and routing.

Conventions

Network

Network	Docker Name	Purpose
Traefik network	traefik-net	Shared external network. Services join as tnet so Traefik can discover them.

Every compose file that needs to be reachable through Traefik must include:

networks:
  tnet:
    name: traefik-net
    external: true

And the service must list tnet under its networks key.

Storage Paths (Host)

Host Path	Purpose
/opt/docker/compose/<service>/	Per-service compose files (managed by Dockge)
/opt/docker/conf/<service>/	Per-service configuration files (bind-mounted into containers)
/tank/	Large / persistent data storage (e.g., AI models, generated images, voice data)

GPU Support

Services requiring GPU access use the NVIDIA Container Toolkit:

runtime: nvidia

or the more explicit device reservation:

deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: all
          capabilities: [gpu]

Homepage Dashboard Labels

Several services include Docker labels for Homepage (a dashboard UI). The convention is:

labels:
  - homepage.group=<Group Name>
  - homepage.name=<Display Name>
  - homepage.icon=<icon identifier>
  - homepage.description=<brief description>
  - homepage.href=http://<host>:<port>

All services reference the VM 102 host IP 10.250.50.70 (PFI-ANA_DOCKER).

Compose File Location

Dockge expects compose files under /opt/docker/compose/ on the Docker host. The live compose files are version-controlled in this project under configs/pfi-ana/docker/compose/. Configuration files that containers bind-mount live under configs/pfi-ana/docker/conf/.

---

Live Services Index

Service	Host Port	Container Port	GPU	Homepage Group	Status	Compose File	Config File
Dockge	5001	5001	No	PFI-ANA	✅ Live	compose/dockge/compose.yaml	—
llama-swap	9292	8080	Yes (CUDA)	—	✅ Live	compose/llama-swap/compose.yaml	conf/llama-swap/config.yaml
ComfyUI	8188	8188	Yes (all caps)	AI Systems	✅ Live	compose/comfyui/compose.yaml	—
VibeVoice	8745	8745	Yes (gpu)	AI Systems	✅ Live	compose/vibevoice/compose.yaml	—
Parakeet STT	8300	8000	Yes (gpu)	AI Systems	✅ Live	compose/parakeet/compose.yaml	—
ChromaDB	8000	8000	No	AI Systems	✅ Live	compose/chromadb/compose.yaml	/opt/docker/conf/chromadb/auth_token

All paths relative to configs/pfi-ana/docker/.

---

Service Details

1. Dockge — Docker Compose Management UI

• Image: louislam/dockge:latest
• Port: 5001 → 5001
• Restart policy: unless-stopped
• Homepage group: PFI-ANA
• Compose file: compose/dockge/compose.yaml

Volumes:

Host / Volume	Container	Purpose
/var/run/docker.sock	/var/run/docker.sock	Docker socket for managing containers
dockge_data (named volume)	/app/data	Dockge application data
/opt/docker/compose	/opt/docker/compose	Compose stack directory

Environment:

• DOCKGE_STACKS_DIR=/opt/docker/compose — tells Dockge where to find/manage compose stacks

Notes: Dockge is the management interface for all other compose stacks on this host. It has full Docker daemon access via the socket mount.

---

2. llama-swap — Multi-Model LLM Gateway

• Image: ghcr.io/mostlygeek/llama-swap:cuda
• Port: 9292 → 8080
• Runtime: nvidia (CUDA)
• Compose file: compose/llama-swap/compose.yaml
• Config file: conf/llama-swap/config.yaml
• Interactive: stdin_open: true, tty: true (required by llama-swap)

Volumes:

Host Path	Container Path	Purpose
/opt/docker/conf/llama-swap/config.yaml	/app/config.yaml	llama-swap configuration (models, groups, params)
/tank/aimodels/llm	/models	LLM model files (GGUF format)

Configured Models (from config.yaml):

Model ID	Display Name	Quantization	Context Size	TTL (s)	Notes
qwen3-4b	Qwen3-4B-Instruct-2507-Q6_K	Q6_K	default	0 (persistent)	Small general-purpose model
glm4.5-air	GLM-4.5-Air Q4_K_M	Q4_K_M	40,000	600	Flash attention enabled
skyfall-r1-31b-q6k	Skyfall 31B v4	Q6_K_L	40,000	600	Flash attention, full GPU offload
GLM-Steam-106B-QK4M-A12B	GLM-Steam 106B A12B	Q4_K_M	40,000	600	2-shard model, MoE with 12B active
kimik2-q2kxl	Kimi K2 Instruct	UD-Q2_K_XL	default	600	8-shard model, only 2 GPU layers (CPU-heavy)
qwen3-coder-30b-iq4-nl	Qwen3 Coder 30B A3B	IQ4_NL	40,000	0 (persistent)	MoE 3B active, coding-optimized
unsloth-granite-4-small	Granite 4.0 Small	Q4_K_M	120,000	0 (persistent)	IBM Granite, deterministic (temp=0)
qwen3.5-35-a3b	Qwen 3.5 35B A3B	UD-Q4_K_XL	32,768	0 (persistent)	MoE, thinking mode, temp=1.0
qwen3.5-35-a3b-code	Qwen 3.5 35B A3B Code	UD-Q4_K_XL	32,768	0 (persistent)	Same model, code-tuned params (temp=0.6)
gemma4-26b-a4b	Gemma 4 26B A4B	UD-Q4_K_XL	32,768	600	MoE 4B active, thinking enabled, supports images
gemma4-31b-dense	Gemma 4 31B Dense	UD-Q4_K_XL	32,768	600	Full dense model, thinking enabled, supports images
embeddinggemma-300M	Embedding Gemma 300M	Q8_0	2,048	0 (persistent)	Embedding model, cls pooling
qwen3-embedding-0.6B	Qwen3 Embedding 0.6B	Q8_0	32,768	0 (persistent)	Embedding model, mean pooling
jina-reranker-v3-0.6B	Jina Reranker v3	Q8_0	32,768	0 (persistent)	Reranking model
bge-reranker-v2-m3-0.6B	BGE Reranker v2 m3	Q8_0	32,768	0 (persistent)	Reranking model

Model Groups:

Group	Swap	Exclusive	Persistent	Members
high-reasoning	false	false	—	qwen3.5-35-a3b, qwen3.5-35-a3b-code, gemma4-31b-dense
utility	false	false	✅	embeddinggemma-300M, bge-reranker-v2-m3-0.6B

Global Settings:

• healthCheckTimeout: 1200 seconds (20 minutes) — long timeout for large models
• logLevel: info
• metricsMaxInMemory: 1000

---

3. ComfyUI — Image Generation UI

• Image: mmartial/comfyui-nvidia-docker:ubuntu24_cuda13.0-latest
• Port: 8188 → 8188
• Runtime: nvidia with full device reservation (gpu, compute, utility capabilities)
• Restart policy: unless-stopped
• Homepage group: AI Systems
• Compose file: compose/comfyui/compose.yaml

Volumes:

Host Path	Container Path	Purpose
/tank/comfy/run	/comfy/mnt	ComfyUI workspace / output directory
/tank/aimodels/img/comfy	/basedir	Image models and ComfyUI base directory

Environment:

Variable	Value	Purpose
WANTED_UID	1001	Run as user ID 1001
WANTED_GID	1002	Run as group ID 1002
BASE_DIRECTORY	/basedir	ComfyUI base directory path
SECURITY_LEVEL	weak	Relaxed security (private network)
NVIDIA_VISIBLE_DEVICES	all	Expose all GPUs
NVIDIA_DRIVER_CAPABILITIES	all	Enable all GPU capabilities

Notes: Runs with user-mapped permissions (UID 1001 / GID 1002). The basedir points to the image model storage on /tank.

---

4. VibeVoice — Voice/Audio AI Service

• Image: eworkerinc/vibevoice:latest
• Container name: vibevoice
• Port: 8745 → 8745
• GPU: Yes (all devices, gpu capability)
• Restart policy: unless-stopped
• Homepage group: AI Systems
• Compose file: compose/vibevoice/compose.yaml

Volumes:

Host Path	Container Path	Purpose
/tank/vibevoice/hf	/root/.cache/huggingface	HuggingFace model cache
/tank/vibevoice/voices	/app/voices	Voice data / presets
/tank/vibevoice/state	/var/lib/eworker	Application state persistence

Environment:

Variable	Value	Purpose
ENABLE_1_5B	true	Enable 1.5B parameter voice model
ENABLE_LARGE	true	Enable large voice model
AUTH_REQUIRED	true	Require authentication
CORS_ENABLED	true	Enable CORS headers
ALLOWED_ORIGINS	*	Allow all origins (development/private network)

---

5. Parakeet STT — Speech-to-Text Service

• Image: parakeet-stt (locally built)
• Port: 8300 → 8000
• GPU: Yes (all devices, gpu capability)
• Restart policy: unless-stopped
• Homepage group: AI Systems
• Compose file: compose/parakeet/compose.yaml
• Env file: .env (not tracked in project — likely contains API keys or model config)

Volumes:

Volume	Container Path	Purpose
parakeet_cache (named volume)	/root/.cache	Model download cache

Notes: Uses a locally-built image (no registry prefix). The .env file is referenced but not stored in the project — it likely contains environment-specific configuration on the Docker host.

---

6. ChromaDB — Vector Database

• Image: chromadb/chroma:latest
• Container name: chromadb
• Port: 8000 → 8000
• GPU: No
• Restart policy: unless-stopped
• Homepage group: AI Systems
• Compose file: compose/chromadb/compose.yaml

Volumes:

Host Path	Container Path	Mode	Purpose
/opt/docker/conf/chromadb	/conf	read-only	Config directory (contains auth_token)
/tank/chromadb	/data	read-write	Persistent vector data

Environment:

Variable	Value	Purpose
CHROMA_SERVER_AUTHN_CREDENTIALS_FILE	/conf/auth_token	Path to auth token file inside container
CHROMA_SERVER_AUTHN_PROVIDER	chromadb.server.auth.token.TokenAuthenticationServerProvider	Enable token-based authentication
IS_PERSISTENT	TRUE	Enable persistent storage
PERSIST_DIRECTORY	/data	Where vector data is stored inside container
ANONYMIZED_TELEMETRY	FALSE	Disable telemetry

Health Check:

Setting	Value
Test	curl -f http://localhost:8000/api/v1/health
Interval	30s
Timeout	10s
Retries	3
Start period	40s

Authentication: Token-based. The auth token is stored at /opt/docker/conf/chromadb/auth_token on the host (mode 600), generated with openssl rand -hex 32. Clients must supply this token to access the API.

Notes: CPU-only service (no GPU). The compose-examples directory contains a reference compose file with Traefik labels and a demo app — see chromadb-setup.md for full deployment instructions and the demo.

---

Infrastructure Summary

┌──────────────────────────────────────────────────────────────┐
│                    PFI-ANA (10.250.50.70)               │
│                   Docker Host (PFI-ANA_DOCKER)          │
│                                                         │
│  ┌──────────┐    Manages all compose stacks              │
│  │  Dockge  │◄─── /opt/docker/compose/*                  │
│  │  :5001   │    /var/run/docker.sock                    │
│  └──────────┘                                           │
│                                                         │
│  ┌──────────────────┐   GPU via passthrough          │
│  │  llama-swap  :9292 (CUDA, multi-model gateway)   │   │
│  │  ├── 15 models (chat, code, embedding, reranker) │   │
│  │  ├── 2 groups (high-reasoning, utility)          │   │
│  │  └── Models from /tank/aimodels/llm              │   │
│  └──────────────────┘   GPU via passthrough          │
│                                                         │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  GPU via      │
│  │  ComfyUI     │  │  VibeVoice   │  │  Parakeet    │  passthrough  │
│  │  :8188       │  │  :8745       │  │  :8300       │              │
│  │  (GPU, img)  │  │  (GPU, voice)│  │  (GPU, STT)  │              │
│  └──────────┘  └──────────┘  └──────────┘              │
│                                                         │
│  ┌──────────┐                                       │
│  │  ChromaDB    │  Token auth, persistent vectors        │
│  │  :8000       │  /tank/chromadb (data)                 │
│  │  (CPU only)  │  /opt/docker/conf/chromadb (config)    │
│  └──────────┘                                       │
│                                                         │
│   ── All services on traefik-net (external) ──         │
│   ── /tank/* = persistent large data storage ──        │
│   ── /opt/docker/* = config + compose files ──        │
└──────────────────────────────────────────────────────────────┘

Port Allocation

Port	Service	Protocol
5001	Dockge	HTTP
8000	ChromaDB	HTTP
8188	ComfyUI	HTTP
8300	Parakeet STT	HTTP (→ container 8000)
8745	VibeVoice	HTTP
9292	llama-swap	HTTP (→ container 8080)

Example Configurations

Reference/example compose files (not live) are stored in configs/pfi-ana/docker/compose-examples/:

• compose-examples/llama-swap/docker-compose.yml — earlier llama-swap reference
• compose-examples/chromadb/ — ChromaDB with Traefik labels, auth, health checks, and demo app

Full ChromaDB setup instructions: chromadb-setup.md