vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: PFI fleet inventory, stacks, tooling, and backup pipeline

Browse Source 
Captures the full workspace state built up to this point:

  - CLAUDE.md + README.md describing conventions and the four-host fleet
    (ana-ml2, ana-docker, nh3-docker, esh-docker-vm).
  - Per-host notes under servers/<host>/ with ssh-target fallback files
    and latest system-details snapshots (two in-compose credential leaks
    scrubbed; the upstream compose files still need to move those to .env).
  - scripts/: server_inspect.sh (read-only remote diagnostic),
    refresh-server-info.sh (dir-driven discovery + snapshot capture with
    validation warnings), add-host.sh, sync-stacks.sh (pull
    compose/conf trees), deploy-stack.sh (push with per-file diff + prompt).
  - stacks/: canonical compose for backrest, beszel, dozzle, llama-swap,
    rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired
    infinity reference. All use the .env-driven + traefik-net + homepage
    label pattern.
  - configs/restic/ana-docker/: first resticprofile config + pre-backup
    hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates
    for the other three hosts to come.
  - docs/pfi/: general infrastructure reference carried over.
  - .gitignore excludes .env, stacks-mirror/, and assorted secret/state
    filenames to prevent re-leaks on later commits.
This commit is contained in:
vh
2026-04-20 14:29:48 -07:00
commit e376d0aec9
55 changed files with 9101 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/backrest/.env.example
+17
View File
@@ -0,0 +1,17 @@
# backrest stack tunables. Copy to `.env` on ana-docker before deploying.
#
# cp .env.example .env
# # edit if needed
# docker compose up -d
#
# Repos, S3 endpoints, restic passwords, and schedules are all configured
# inside the Backrest web UI after first boot — nothing belongs here.
# Image version — pin for reproducibility (`latest` for edge)
BACKREST_VERSION=latest
# Host port for the web UI (container listens on 9898 internally)
BACKREST_PORT=9898
# Timezone — affects scheduler display and log timestamps
TZ=America/Los_Angeles
stacks/backrest/README.md
+65
View File
@@ -0,0 +1,65 @@
# backrest
Web UI over restic repositories. Runs **once, on ana-docker**, and points at every per-host restic repo on both site-local S3 endpoints to give a single pane of glass for snapshot history, restores, and alerts.
**Server:** ana-docker
**Port:** `http://10.250.50.70:9898`
## Role in the fleet
- Actual backups are run by per-host `systemd` timers calling `restic` (see the backup design; not yet deployed). Each host writes to its site-local S3 bucket (TrueNAS on ana, Synology on nh3).
- **This container does not run backups by default** — it's a viewer/manager pointed at existing repos. (Backrest *can* be the scheduler instead of systemd timers; we're keeping the scheduler on the host for simplicity.)
- Because it only needs to talk to S3 endpoints (not host filesystems), no bind mounts of host paths are required.
## Deploy
```bash
# On ana-docker:
sudo mkdir -p /opt/docker/compose/backrest
sudo chown $USER /opt/docker/compose/backrest
cd /opt/docker/compose/backrest
# scp compose.yaml + .env.example from this workspace, then:
cp .env.example .env
# edit .env if you want a different port or TZ
docker compose config
docker compose up -d
docker compose logs -f
```
Open `http://10.250.50.70:9898` and set the admin credentials on first load.
## First-time configuration (in the UI)
For each per-host repo (to be added once the restic pipeline is running):
1. **Add Repository** → fill in:
- **ID:** e.g. `ana-docker`, `ana-ml2`, `nh3-docker`, `esh-docker-vm`
- **URI:** `s3:https://10.250.50.50:4521/pfi-backups/ana/ana-docker/` (or the nh3 endpoint, depending on host)
- **Password:** the restic passphrase for that host
- **Env:** `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` for the S3 endpoint
2. *(Do not create a plan unless you want Backrest to drive the schedule — leave schedules to the systemd timers on each host.)*
3. Verify: the repo should list snapshots pulled by the host's systemd timer within a few minutes.
## Backup scope, when wired up
- ana-docker, ana-ml2 → TrueNAS S3 at `10.250.50.50:4521`
- nh3-docker, esh-docker-vm → Synology S3 at `10.100.50.50:4521`
- Cross-site rclone sync makes each bucket also hold the other site's data, so you can restore *either* host from *either* side if one NAS is down.
## Scaling knobs
- **Upgrade:** `docker compose pull && docker compose up -d`.
- **Backup of Backrest itself:** its config/DB lives in the `backrest_config` and `backrest_data` volumes — include those in ana-docker's restic plan so recreating the UI doesn't mean re-entering every repo.
## Why not `offen/docker-volume-backup`?
Two instances on `esh-docker-vm` (paperless-ngx, pgadmin) currently use the `offen/docker-volume-backup` sidecar pattern. Those will be retired once restic is in place:
- No encryption — backups sit in the clear on NFS.
- No dedup — every run writes a full tarball; storage grows linearly.
- Per-stack config — every new service needs its own sidecar wiring.
- No cross-host index — restores require knowing which tarball lives where.
Restic + Backrest solves all four at the cost of one extra binary per host. Leave the sidecars running until the restic plan is verified, then remove them in a scheduled change.
stacks/backrest/compose.yaml
+58
View File
@@ -0,0 +1,58 @@
# Backrest — web UI over restic repositories.
#
# Role here: single central viewer for every host's restic repo on both
# site-local S3 endpoints (TrueNAS at ana, Synology at nh3). Per-host
# `restic` runs will still be driven by systemd timers on each host; this
# stack is how we see what ran, browse snapshots, and restore.
#
# Repos and S3 credentials are configured in the Backrest UI after first
# boot — nothing baked into this file. Data (its own SQLite + queue) lives
# in a named volume so the config survives container recreation.
#
# All tunables live in .env — edit that, not this file.
services:
backrest:
image: garethgeorge/backrest:${BACKREST_VERSION}
container_name: backrest
hostname: backrest
restart: unless-stopped
ports:
- "${BACKREST_PORT}:9898"
volumes:
- backrest_data:/data
- backrest_config:/config
- backrest_cache:/cache
- backrest_tmp:/tmp
environment:
- BACKREST_DATA=/data
- BACKREST_CONFIG=/config/config.json
- XDG_CACHE_HOME=/cache
- TMPDIR=/tmp
- TZ=${TZ:-America/Los_Angeles}
- BACKREST_PORT=0.0.0.0:9898
healthcheck:
test: ["CMD", "wget", "-qO-", "http://localhost:9898/"]
interval: 30s
timeout: 10s
retries: 3
start_period: 15s
networks:
- tnet
labels:
- homepage.group=PFI-ANA
- homepage.name=Backrest
- homepage.icon=mdi-backup-restore
- homepage.description=Restic snapshot viewer / restore UI
- homepage.href=http://10.250.50.70:${BACKREST_PORT}
volumes:
backrest_data:
backrest_config:
backrest_cache:
backrest_tmp:
networks:
tnet:
name: traefik-net
external: true
stacks/beszel/.env.example
+43
View File
@@ -0,0 +1,43 @@
# beszel stack tunables. Copy to `.env` on each server before deploying.
#
# cp .env.example .env
# # edit for this host
# docker compose up -d
#
# Same compose.yaml on both servers — COMPOSE_PROFILES picks the role.
# Image version — pin for reproducibility (`latest` for edge)
BESZEL_VERSION=latest
# ------------------------------------------------------------------------
# On ana-docker (hub + local agent):
# COMPOSE_PROFILES=hub,agent
# BESZEL_PORT=8090
# BESZEL_HUB_KEY=<copy from hub "Add system" dialog after first boot>
#
# On ana-ml2 (agent only):
# COMPOSE_PROFILES=agent
# BESZEL_HUB_KEY=<same key as above>
# ------------------------------------------------------------------------
COMPOSE_PROFILES=hub,agent
# ---- Hub-only -----------------------------------------------------------
# Host port for the web UI (container listens on 8090 internally).
BESZEL_PORT=8090
# ---- Agent-only ---------------------------------------------------------
# Host port the agent listens on. The hub SSHes into agents over this port.
BESZEL_AGENT_PORT=45876
# Hub's SSH public key — paste from the hub UI on first run.
# Grab it by clicking "Add System" → copy the key shown in the dialog.
BESZEL_HUB_KEY=
# Extra filesystems to track beyond the root mount, comma-separated.
# Examples:
# on ana-ml2: /tank
# on ana-docker: /mnt/backup,/mnt/compose
BESZEL_EXTRA_FS=
stacks/beszel/README.md
+88
View File
@@ -0,0 +1,88 @@
# beszel
Lightweight monitoring — CPU, memory, disk, network, and per-container stats for every Docker host, with alerts over email/webhook. Pairs with Dozzle (logs) on the same server.
**Deploys to:**
- **ana-docker** (hub + local agent) — UI at `http://10.250.50.70:8090`
- **ana-ml2** (agent only) — listens on `10.250.50.54:45876`
- **nh3-docker** (agent only, cross-site) — listens on `10.100.50.40:45876`
Same compose.yaml on each host. Per-host `.env` sets `COMPOSE_PROFILES` to bring up the right combination. Each agent host is added individually in the hub UI.
## How hub ↔ agent auth works
Beszel uses SSH-key-based auth: the hub generates its own keypair on first boot, and each agent must be seeded with the hub's **public key** via the `KEY` env var. Agents listen on a port (default 45876); the hub pulls metrics by connecting to them with that key.
Operator flow on first deploy:
1. Bring up the **hub** on ana-docker with `BESZEL_HUB_KEY=` blank and the agent profile disabled.
2. Open the UI, create the admin account, click **Add System** — Beszel shows the public key.
3. Copy the key into `BESZEL_HUB_KEY` in the `.env` on both hosts.
4. Re-deploy the hub with `COMPOSE_PROFILES=hub,agent` to add the local agent; deploy the agent on ana-ml2.
5. Back in the UI, **Add System** with `host=127.0.0.1 port=45876` (local) and `host=10.250.50.54 port=45876` (ana-ml2).
## Deploy — hub + local agent (ana-docker)
```bash
ssh ana-docker
sudo mkdir -p /opt/docker/compose/beszel
sudo chown $USER /opt/docker/compose/beszel
cd /opt/docker/compose/beszel
# scp compose.yaml + .env.example, then:
cp .env.example .env
# First pass — hub only, no key yet:
# COMPOSE_PROFILES=hub
# BESZEL_PORT=8090
docker compose up -d
# Open http://10.250.50.70:8090 → create admin → click "Add System" →
# copy the displayed public key into BESZEL_HUB_KEY.
# Second pass — add the local agent:
# COMPOSE_PROFILES=hub,agent
# BESZEL_EXTRA_FS=/mnt/backup,/mnt/compose
docker compose up -d
```
## Deploy — agent (ana-ml2)
```bash
ssh ana-ml2
sudo mkdir -p /opt/docker/compose/beszel
sudo chown $USER /opt/docker/compose/beszel
cd /opt/docker/compose/beszel
# scp the same compose.yaml + .env.example, then:
cp .env.example .env
# Edit to:
# COMPOSE_PROFILES=agent
# BESZEL_HUB_KEY=<same key as the hub>
# BESZEL_EXTRA_FS=/tank
docker compose up -d
docker compose logs -f
```
Then in the hub UI, **Add System** with `host=10.250.50.54`, `port=45876`.
## Verify
```bash
# Hub health
curl -s http://10.250.50.70:8090/api/health
# Agent reachable
ssh ana-docker 'nc -zv 10.250.50.54 45876'
# Local agent reachable from hub container
docker exec beszel nc -zv host.docker.internal 45876
```
## Sizing / impact
The agent is ~10 MB RAM and negligible CPU — runs fine alongside anything on ana-ml2 including GPU workloads. Host-mode networking means it has no port conflicts with other stacks as long as `BESZEL_AGENT_PORT` stays unique.
## Alerts
Configured inside the hub UI (Settings → Notifications). Supports email (SMTP), Gotify, ntfy, Discord, Slack, and generic webhooks. Alert rules attach to per-system or global thresholds (CPU, memory, disk, container down, etc.).
stacks/beszel/compose.yaml
+63
View File
@@ -0,0 +1,63 @@
# Beszel — lightweight server/container monitoring.
#
# Hub: single web UI with the SQLite store. Agents: per-host metric collectors
# that the hub pulls from over SSH.
#
# Multi-host layout via compose profiles:
# COMPOSE_PROFILES=hub → hub only (ana-docker)
# COMPOSE_PROFILES=hub,agent → hub + local agent on the same host
# COMPOSE_PROFILES=agent → agent only (ana-ml2)
#
# The agent uses network_mode: host so it sees real host CPU/mem/net/disk
# counters rather than container-scoped ones — that's why it can't share
# the tnet network with the hub.
#
# All tunables live in .env — edit that, not this file.
services:
beszel:
image: henrygd/beszel:${BESZEL_VERSION}
container_name: beszel
profiles: [hub]
restart: unless-stopped
ports:
- "${BESZEL_PORT}:8090"
volumes:
- beszel_data:/beszel_data
healthcheck:
test: ["CMD", "wget", "-qO-", "http://localhost:8090/api/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 15s
networks:
- tnet
labels:
- homepage.group=PFI-ANA
- homepage.name=Beszel
- homepage.icon=mdi-chart-line
- homepage.description=Server + container monitoring
- homepage.href=http://10.250.50.70:${BESZEL_PORT}
beszel-agent:
image: henrygd/beszel-agent:${BESZEL_VERSION}
container_name: beszel-agent
profiles: [agent]
restart: unless-stopped
network_mode: host
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- beszel_agent_data:/var/lib/beszel-agent
environment:
- PORT=${BESZEL_AGENT_PORT:-45876}
- KEY=${BESZEL_HUB_KEY}
- EXTRA_FILESYSTEMS=${BESZEL_EXTRA_FS:-}
volumes:
beszel_data:
beszel_agent_data:
networks:
tnet:
name: traefik-net
external: true
stacks/dozzle/.env.example
+50
View File
@@ -0,0 +1,50 @@
# dozzle stack tunables. Copy to `.env` on each server before deploying.
#
# cp .env.example .env
# # edit for this host
# docker compose up -d
#
# Same compose.yaml on both servers — COMPOSE_PROFILES picks the role.
# Image version — pin for reproducibility (`latest` for edge)
DOZZLE_VERSION=latest
# ------------------------------------------------------------------------
# On ana-docker (hub):
# COMPOSE_PROFILES=hub
# DOZZLE_HOSTNAME=ana-docker
# DOZZLE_REMOTE_AGENT=10.250.50.54:7007
#
# On ana-ml2 (agent):
# COMPOSE_PROFILES=agent
# DOZZLE_HOSTNAME=ana-ml2
# ------------------------------------------------------------------------
COMPOSE_PROFILES=hub
# Display name for this host in the UI (shown as a tab / section header).
DOZZLE_HOSTNAME=ana-docker
# ---- Hub-only -----------------------------------------------------------
# Host port for the web UI (container listens on 8080 internally).
DOZZLE_PORT=8088
# Comma-separated list of remote agents the hub should connect to.
# Leave blank if this host only views its own containers.
DOZZLE_REMOTE_AGENT=10.250.50.54:7007,10.100.50.40:7007
# Auth — `none` is fine behind the LAN / a reverse proxy with auth.
# Switch to `simple` and set USERNAME/PASSWORD to gate the UI itself.
DOZZLE_AUTH_PROVIDER=none
DOZZLE_USERNAME=
DOZZLE_PASSWORD=
# ---- Agent-only ---------------------------------------------------------
# Host port the agent listens on (container listens on 7007 internally).
DOZZLE_AGENT_PORT=7007
# Bind address — restrict to the LAN interface if you want belt-and-braces
# beyond what the firewall already enforces. Default 0.0.0.0 exposes on all.
DOZZLE_AGENT_BIND=0.0.0.0
stacks/dozzle/README.md
+74
View File
@@ -0,0 +1,74 @@
# dozzle
Container log viewer. One UI on **ana-docker** aggregates logs from every Docker host via remote agents.
**Deploys to:**
- **ana-docker** (hub) — UI at `http://10.250.50.70:8088`
- **ana-ml2** (agent) — listens on `10.250.50.54:7007`
- **nh3-docker** (agent, cross-site) — listens on `10.100.50.40:7007`
One compose.yaml lives on each host. The per-host `.env` sets `COMPOSE_PROFILES=hub` or `COMPOSE_PROFILES=agent` so `docker compose up -d` brings up the right service. On the hub, add every agent to `DOZZLE_REMOTE_AGENT` as a comma-separated list (e.g. `10.250.50.54:7007,10.100.50.40:7007`).
## Auth / TLS note
Dozzle agents and hub auto-generate mTLS certificates on first run. On the trusted LAN (10.250.0.0/16) the default config is fine. If you ever expose an agent beyond the LAN, generate and pin certificates explicitly per the Dozzle docs (`dozzle generate`). The web UI itself is unauthenticated by default — flip `DOZZLE_AUTH_PROVIDER=simple` and set `DOZZLE_USERNAME`/`DOZZLE_PASSWORD` in the hub `.env` if you want a login gate.
## Deploy — hub (ana-docker)
```bash
ssh ana-docker
sudo mkdir -p /opt/docker/compose/dozzle
sudo chown $USER /opt/docker/compose/dozzle
cd /opt/docker/compose/dozzle
# scp compose.yaml + .env.example from this workspace, then:
cp .env.example .env
# Ensure:
# COMPOSE_PROFILES=hub
# DOZZLE_HOSTNAME=ana-docker
# DOZZLE_REMOTE_AGENT=10.250.50.54:7007
# DOZZLE_PORT=8088
docker compose config
docker compose up -d
docker compose logs -f
```
## Deploy — agent (ana-ml2)
```bash
ssh ana-ml2
sudo mkdir -p /opt/docker/compose/dozzle
sudo chown $USER /opt/docker/compose/dozzle
cd /opt/docker/compose/dozzle
# scp the same compose.yaml + .env.example, then:
cp .env.example .env
# Edit to:
# COMPOSE_PROFILES=agent
# DOZZLE_HOSTNAME=ana-ml2
# DOZZLE_AGENT_PORT=7007
docker compose config
docker compose up -d
docker compose logs -f
```
## Verify
```bash
# Hub health (from anywhere on LAN)
curl -s http://10.250.50.70:8088/healthz
# Agent reachable from the hub's perspective
ssh ana-docker 'nc -zv 10.250.50.54 7007'
# Open http://10.250.50.70:8088 — you should see two tabs:
# "ana-docker" (local containers) and "ana-ml2" (via agent).
```
## Troubleshooting
- **Hub shows only local containers:** agent is unreachable. Check firewall rules on ana-ml2 (port 7007 must be open from 10.250.50.70) and that the agent is actually listening (`ss -tlnp | grep 7007`).
- **Agent keeps restarting:** verify the docker.sock bind mount is read-only and the socket exists.
- **Certificate mismatch after image upgrade:** delete the `dozzle_data` and `dozzle_agent_data` volumes on both hosts and redeploy to regenerate.
stacks/dozzle/compose.yaml
+66
View File
@@ -0,0 +1,66 @@
# Dozzle — container log viewer.
#
# Multi-host layout via compose profiles:
# COMPOSE_PROFILES=hub → runs the web UI (deploy on ana-docker)
# COMPOSE_PROFILES=agent → runs the remote agent (deploy on ana-ml2)
#
# Same compose.yaml on both servers; per-host `.env` picks the profile.
#
# All tunables live in .env — edit that, not this file.
services:
dozzle:
image: amir20/dozzle:${DOZZLE_VERSION}
container_name: dozzle
profiles: [hub]
restart: unless-stopped
ports:
- "${DOZZLE_PORT}:8080"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- dozzle_data:/data
environment:
- DOZZLE_HOSTNAME=${DOZZLE_HOSTNAME}
- DOZZLE_REMOTE_AGENT=${DOZZLE_REMOTE_AGENT:-}
- DOZZLE_AUTH_PROVIDER=${DOZZLE_AUTH_PROVIDER:-none}
- DOZZLE_USERNAME=${DOZZLE_USERNAME:-}
- DOZZLE_PASSWORD=${DOZZLE_PASSWORD:-}
healthcheck:
test: ["CMD", "/dozzle", "healthcheck"]
interval: 30s
timeout: 10s
retries: 3
start_period: 15s
networks:
- tnet
labels:
- homepage.group=PFI-ANA
- homepage.name=Dozzle
- homepage.icon=mdi-text-box-search
- homepage.description=Container logs (ana-docker + ana-ml2)
- homepage.href=http://10.250.50.70:${DOZZLE_PORT}
dozzle-agent:
image: amir20/dozzle:${DOZZLE_VERSION}
container_name: dozzle-agent
profiles: [agent]
restart: unless-stopped
command: agent
ports:
- "${DOZZLE_AGENT_BIND:-0.0.0.0}:${DOZZLE_AGENT_PORT}:7007"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- dozzle_agent_data:/data
environment:
- DOZZLE_HOSTNAME=${DOZZLE_HOSTNAME}
networks:
- tnet
volumes:
dozzle_data:
dozzle_agent_data:
networks:
tnet:
name: traefik-net
external: true
stacks/infinity/.env.example
+30
View File
@@ -0,0 +1,30 @@
# Infinity stack tunables. Copy this to `.env` on the server before deploying.
#
# cp .env.example .env
# # edit .env with real values
# docker compose up -d
# Image version — pin for reproducibility (`latest` for edge)
INFINITY_VERSION=latest
# Port exposed on host
INFINITY_PORT=7997
# GPU assignment (ana-ml2 has 0 and 1; default 1 keeps 0 free for heavy LLM work)
GPU_ID=1
# Models — both served simultaneously; reference by the full repo name in requests
EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B
RERANK_MODEL=Qwen/Qwen3-Reranker-0.6B
# Inference engine: torch (widest support) or optimum (ONNX, sometimes faster)
ENGINE=torch
# Batch size — 32 is a safe default; bump for throughput if VRAM allows
BATCH_SIZE=32
# Optional API key — leave blank for no auth (fine on the internal network)
API_KEY=
# HuggingFace token — only needed for gated models
HF_TOKEN=
stacks/infinity/README.md
+65
View File
@@ -0,0 +1,65 @@
# infinity
OpenAI-compatible embeddings + reranker server. One container serves both embedding and reranker models simultaneously.
**Server:** ana-ml2
**Port:** 7997 (infinity default)
**GPU:** pinned to GPU 1 by default (configurable via `.env`)
## What it replaces / supersedes
- `qwen3-embedding-0.6B` entry in llama-swap (llama.cpp GGUF → infinity transformer)
- `qwen3-reranker-0.6B` entry in llama-swap
Once infinity is verified stable, retire those two entries from `stacks/llama-swap/config.yaml`.
## Deploy
```bash
# On ana-ml2:
sudo mkdir -p /opt/docker/compose/infinity
sudo chown $USER /opt/docker/compose/infinity
cd /opt/docker/compose/infinity
# Copy compose.yaml + .env.example here (e.g. via scp from this workspace)
# Then:
cp .env.example .env
# edit .env — pick GPU, models, etc.
# Pre-download models into the shared HF cache (optional, speeds first boot)
HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^EMBED_MODEL .env | cut -d= -f2)"
HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^RERANK_MODEL .env | cut -d= -f2)"
# Dry-parse
docker compose config
# Launch
docker compose up -d
docker compose logs -f
```
## Verify
```bash
# Health
curl -s http://localhost:7997/health
# Embedding
curl -s http://localhost:7997/embeddings \
-H "Content-Type: application/json" \
-d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":["hello world"]}' | jq .
# Reranker
curl -s http://localhost:7997/rerank \
-H "Content-Type: application/json" \
-d '{"model":"Qwen/Qwen3-Reranker-0.6B","query":"what is a cat","documents":["cats are mammals","dogs bark"]}' | jq .
# Listed models
curl -s http://localhost:7997/models | jq .
```
## Scaling knobs
- **`BATCH_SIZE`** in `.env` — bigger = higher throughput, more VRAM. 32 is safe; try 64 or 128 if you have headroom.
- **Model size** — Qwen3-Embedding/Reranker come in 0.6B / 4B / 8B. Pick based on quality-vs-latency tradeoff. On RTX 6000 Ada 46 GB, the 8B pair fits easily (~20 GB VRAM).
- **`ENGINE=optimum`** — uses ONNX runtime, sometimes faster. Requires the model to have ONNX weights available; fall back to `torch` if it errors on startup.
stacks/infinity/compose.yaml
+63
View File
@@ -0,0 +1,63 @@
# Infinity — OpenAI-compatible embeddings + reranker server.
#
# Serves embedding and reranker models simultaneously from one container
# on port 7997 (HTTP). Consumers: AIPA agents (search/retrieval), LibreChat
# RAG, anything that needs vector embeddings.
#
# All tunables live in .env — edit that, not this file.
#
# Pre-download models to avoid first-run delay:
# HF_HOME=/tank/aimodels/huggingface hf download Qwen/Qwen3-Embedding-0.6B
# HF_HOME=/tank/aimodels/huggingface hf download Qwen/Qwen3-Reranker-0.6B
services:
infinity:
image: michaelf34/infinity:${INFINITY_VERSION}
container_name: infinity
restart: unless-stopped
ports:
- "${INFINITY_PORT}:7997"
volumes:
- /tank/aimodels/huggingface:/hfcache
environment:
- HF_HOME=/hfcache
- HF_HUB_CACHE=/hfcache/hub
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
command: >
v2
--model-id ${EMBED_MODEL}
--model-id ${RERANK_MODEL}
--engine ${ENGINE}
--device cuda
--batch-size ${BATCH_SIZE}
--host 0.0.0.0
--port 7997
--api-key ${API_KEY:-}
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids:
- "${GPU_ID}"
capabilities:
- gpu
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:7997/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 120s
networks:
- tnet
labels:
- homepage.group=AI Systems
- homepage.name=Infinity
- homepage.icon=mdi-vector-arrange-below
- homepage.description=Embeddings + Reranker API (ana-ml2)
- homepage.href=http://10.250.50.54:7997/docs
networks:
tnet:
name: traefik-net
external: true
stacks/llama-swap/.env.example
+18
View File
@@ -0,0 +1,18 @@
# llama-swap stack tunables. Copy to `.env` on ana-docker before deploying.
#
# cp .env.example .env
# # edit if needed
# docker compose up -d
# Image tag. `cuda` is the CUDA-enabled build; pin to a specific release
# (e.g. `cuda-v0.0.6`) for reproducibility once upstream tags stabilize.
LLAMA_SWAP_VERSION=cuda
# Host port for the OpenAI-compatible API (container listens on 8080)
LLAMA_SWAP_PORT=9292
# Host paths
# --- Legacy GGUF models referenced by config.yaml as `-m /models/<file>`
MODELS_DIR=/tank/aimodels/llm
# --- Shared HuggingFace cache used by `-hf` model entries
HF_CACHE_DIR=/tank/aimodels/huggingface
stacks/llama-swap/README.md
+52
View File
@@ -0,0 +1,52 @@
# llama-swap
GGUF model server with on-demand model swapping. Served via llama.cpp's `llama-server` under the llama-swap proxy.
**Server:** ana-ml2
**Port:** 9292 (configurable via `.env`)
**GPU:** both (unpinned — `runtime: nvidia` grants access to all devices; per-model GPU selection happens inside `config.yaml`)
## Files
- **`compose.yaml`** — canonical compose. Deployed to `/opt/docker/compose/llama-swap/compose.yaml` on ana-ml2.
- **`.env.example`** — template for the per-host `.env`. Copy to `.env` on the server and tweak.
- **`config.yaml`** — model definitions and groups. Deployed to `/opt/docker/conf/llama-swap/config.yaml` on the server.
Homepage labels are in the compose file under the `AI Systems` group, matching the convention used by `vllm-qwen3` and `infinity`.
## Deploy a fresh install
```bash
scripts/deploy-stack.sh ana-ml2 llama-swap
ssh ana-ml2 '
cd /opt/docker/compose/llama-swap && \
cp -n .env.example .env && \
docker compose config && \
docker compose up -d && \
docker compose logs --tail=30
'
```
## Model reference conventions
- **Modern entries:** use `-hf <user>/<repo>[:<quant>]` — reads from the shared HF cache, nothing to pre-stage outside `hf download`
- **Legacy entries:** use `--model /models/<dir>/<file>.gguf` — reads GGUFs from `/tank/aimodels/llm/` (pre-HF-cache era, gradually being migrated)
New models should prefer the `-hf` pattern.
## Deploy updates to config only
```bash
# After editing config.yaml here:
scp config.yaml ana-ml2:/opt/docker/conf/llama-swap/config.yaml
ssh ana-ml2 'cd /opt/docker/compose/llama-swap && docker compose restart'
```
## Deploy updates to compose only
```bash
# After editing compose.yaml or .env.example here:
scripts/deploy-stack.sh ana-ml2 llama-swap
ssh ana-ml2 'cd /opt/docker/compose/llama-swap && docker compose up -d'
```
stacks/llama-swap/compose.yaml
+50
View File
@@ -0,0 +1,50 @@
# llama-swap — GGUF model server with on-demand model swapping.
#
# Proxies OpenAI-compatible API requests to llama.cpp server instances
# and swaps which model is loaded into VRAM per request. Runs on
# ana-ml2 using both GPUs dynamically (no explicit device pinning —
# llama-swap picks per-model-definition).
#
# Model definitions live in /opt/docker/conf/llama-swap/config.yaml on
# the server. Canonical copy of that config is config.yaml in this
# workspace; deploy with scp + `docker compose restart` or the script
# at the bottom of README.md.
#
# All tunables live in .env — edit that, not this file.
services:
llama-swap:
image: ghcr.io/mostlygeek/llama-swap:${LLAMA_SWAP_VERSION}
container_name: llama-swap
restart: unless-stopped
stdin_open: true
tty: true
runtime: nvidia
ports:
- "${LLAMA_SWAP_PORT}:8080"
volumes:
- /opt/docker/conf/llama-swap/config.yaml:/app/config.yaml
- ${MODELS_DIR}:/models
- ${HF_CACHE_DIR}:/hfcache
environment:
- HF_HOME=/hfcache
- HF_HUB_CACHE=/hfcache/hub
healthcheck:
test: ["CMD-SHELL", "curl -fsS http://localhost:8080/ >/dev/null || exit 1"]
interval: 30s
timeout: 10s
retries: 3
start_period: 30s
networks:
- tnet
labels:
- homepage.group=AI Systems
- homepage.name=llama-swap
- homepage.icon=mdi-swap-horizontal
- homepage.description=GGUF model swapper (llama.cpp; ana-ml2)
- homepage.href=http://10.250.50.54:${LLAMA_SWAP_PORT}
networks:
tnet:
name: traefik-net
external: true
stacks/llama-swap/config.yaml
+463
View File
@@ -0,0 +1,463 @@
# ============================================================================
# llama-swap configuration for PFI-ANA
# Optimized and synchronized with /models disk inventory
# Last updated: 2026-04-10
#
# KB Sources:
# - reference/nemotron-3-super-running-parameters.md
# - reference/nemotron-3-nano-running-parameters.md
# - reference/qwen3.5-running-parameters.md
# - reference/qwen3-coder-next-running-parameters.md
# - reference/gemma-4-running-parameters.md
# - reference/qwen3-embedding-running-parameters.md
# - reference/qwen3-reranker-running-parameters.md
#
# Changelog:
# 2025-07-22: Removed jina-reranker-v3 (unused, out of rotation).
# 2026-04-10: Fixed Qwen3-Embedding pooling (mean→last; causal LM uses last-token
# pooling). Fixed ctx-size 4096→8192 for embedding+reranker. Fixed
# reranker: removed --embeddings flag (not an embedding model).
# 2026-04-17: Added Qwen3.6-35B-A3B Abliterated Heretic Q8_0 via -hf syntax.
# Requires HF_HOME=/hfcache in compose (see docker-compose.yml).
# New convention: use -hf repo[:quant] instead of --model /path.
# ============================================================================
# Default 1200 seconds (20 min) to wait for model to be available to load.
healthCheckTimeout: 1200
# logLevel: sets the logging value
# - optional, default: info
# - Valid log levels: debug, info, warn, error
logLevel: info
# metricsMaxInMemory: maximum number of metrics to keep in memory
# - optional, default: 1000
metricsMaxInMemory: 1000
# startPort: sets the starting port number for the automatic ${PORT} macro.
# - optional, default: 5800
# - the ${PORT} macro can be used in model.cmd and model.proxy settings
# - it is automatically incremented for every model that uses it
# startPort: 10001
models:
# ==========================================================================
# QWEN 3.5 MODELS (KB-recommended settings)
# - Thinking mode: temp 1.0, top-p 0.95, top-k 20, min-p 0.0, presence_penalty 1.5
# - Coding (precise): temp 0.6, top-p 0.95, top-k 20, min-p 0.0, presence_penalty 0.0
# - Non-thinking general: temp 0.7, top-p 0.8, top-k 20, min-p 0.0, presence_penalty 1.5
# - Context: 256K native (start 16K-32K for responsiveness)
# - Gibberish fix: add --cache-type-k bf16 --cache-type-v bf16
# - No Ollama support for Qwen3.5 GGUFs — use llama.cpp only
# ==========================================================================
"qwen3.5-35-a3b":
name: "Qwen 3.5 35B-A3B Thinking"
description: "MoE reasoning model. 3B active params, general-purpose thinking/chat."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/unsloth_Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 32768
--flash-attn on
--temp 1.0
--top-p 0.95
--top-k 20
--min-p 0.00
--presence-penalty 1.5
--chat-template-kwargs '{"enable_thinking":true}'
"qwen3.5-122b-a10b":
name: "Qwen 3.5 122B-A10B UD-Q4_K_XL"
description: "Large MoE reasoning model. 10B active params, heavy reasoning tasks."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/unsloth_Qwen3.5-122B-A10B-GGUF/UD-Q4_K_XL/Qwen3.5-122B-A10B-UD-Q4_K_XL-00001-of-00003.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 32768
--flash-attn on
--temp 1.0
--top-p 0.95
--top-k 20
--min-p 0.00
--presence-penalty 1.5
--chat-template-kwargs '{"enable_thinking":true}'
"qwen3.5-9b":
name: "Qwen 3.5 9B UD-Q4_K_XL"
description: "Dense 9B model. Lightweight general-purpose chat and reasoning."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/unsloth_Qwen3.5-9B-GGUF/Qwen3.5-9B-UD-Q4_K_XL.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 32768
--flash-attn on
--temp 1.0
--top-p 0.95
--top-k 20
--min-p 0.00
--presence-penalty 1.5
--chat-template-kwargs '{"enable_thinking":true}'
# --------------------------------------------------------------------------
# Qwen 3.6 — uses -hf syntax, reads from HF_HOME=/hfcache (host pre-download)
# --------------------------------------------------------------------------
"qwen3.6-35-a3b-abliterated":
name: "Qwen 3.6 35B-A3B Abliterated Heretic Q8_0"
description: "Qwen3.6 MoE, 3B active. Abliterated/heretic variant of BF16 quantized to Q8_0. ~38GB."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--jinja
-hf IIEleven11/Qwen3.6-35B-A3B-Abliterated-Heretic-BF16-Q8_0-GGUF
--port ${PORT}
--n-gpu-layers 999
--ctx-size 32768
--flash-attn on
--temp 1.0
--top-p 0.95
--top-k 20
--min-p 0.00
--presence-penalty 1.5
--repeat-penalty 1.0
--reasoning on
--reasoning-format deepseek
# ==========================================================================
# NEMOTRON MODELS (KB-recommended settings)
# - General Chat: temp 1.0, top-p 1.0, min_p 0.01
# - Tool Calling: temp 0.6, top-p 0.95, min_p 0.01
# - NoPE architecture: no YaRN needed
# - DEPRECATED --special flag for reasoning tokens
# - --special flag causes issues.
# - Start ctx 16K-32K, increase cautiously
# ==========================================================================
"nemotron-3-super-120b":
name: "NVIDIA Nemotron 3 Super 120B-A12B UD-Q4_K_XL"
description: "Flagship NVIDIA reasoning model. 12B active of 120B, MoE. 64-72GB VRAM at Q4."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/unsloth_NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_XL/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-00001-of-00003.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 16384
--flash-attn on
--temp 1.0
--top-p 1.0
--min-p 0.01
--seed 3407
"nemotron-3-nano-30b":
name: "NVIDIA Nemotron 3 Nano 30B-A3B UD-Q4_K_XL"
description: "Compact Nemotron. 3B active of 30B, MoE. ~24GB at Q4. Best performance/size on 24GB GPUs."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--special
--model /models/unsloth_Nemotron-3-Nano-30B-A3B-GGUF/Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 32768
--flash-attn on
--temp 1.0
--top-p 1.0
--min-p 0.01
--seed 3407
# ==========================================================================
# GEMMA 4 MODELS (KB-recommended settings)
# - All variants: temp 1.0, top-p 0.95, top-k 64, repeat_penalty 1.0
# - Thinking: enable via --chat-template-kwargs '{"enable_thinking":true}'
# - Multi-turn: only keep final visible answer in history (not thought blocks)
# - Context: E2B/E4B=128K, 26B-A4B/31B=256K. Start at 32K.
# - ⚠️ Do NOT use CUDA 13.2 runtime — causes poor outputs
# - Use llama-server (not llama-cli) for thinking control
# ==========================================================================
"gemma4-26b-a4b":
name: "Gemma 4 26B-A4B"
description: "Google DeepMind Gemma 4 MoE, 4B active params, 256K context. Best speed/quality tradeoff."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/unsloth_gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 32768
--flash-attn on
--temp 1.0
--top-p 0.95
--top-k 64
--repeat-penalty 1.0
--chat-template-kwargs '{"enable_thinking":true}'
"gemma4-31b-dense":
name: "Gemma 4 31B Dense"
description: "Google DeepMind Gemma 4 dense 31B. Maximum quality for complex reasoning, 256K context."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/unsloth_gemma-4-31B-it-GGUF/gemma-4-31B-it-UD-Q4_K_XL.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 32768
--flash-attn on
--temp 1.0
--top-p 0.95
--top-k 64
--repeat-penalty 1.0
--chat-template-kwargs '{"enable_thinking":true}'
# ==========================================================================
# GLM MODELS
# ==========================================================================
"glm4.7-flash":
name: "GLM 4.7 Flash UD-Q4_K_XL"
description: "THUDM GLM 4.7 Flash. Fast inference, general-purpose chat."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/unsloth_GLM-4.7-Flash-GGUF/GLM-4.7-Flash-UD-Q4_K_XL.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 40000
--flash-attn on
--temp 0.6
--top-p 0.95
"glm-steam-106b":
name: "GLM Steam 106B-A12B Q4_K_M"
description: "TheDrummer GLM Steam MoE. 12B active of 106B. Creative and RP-focused."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/RP/bartowski_TheDrummer_GLM-Steam-106B-A12B-v1-GGUF/TheDrummer_GLM-Steam-106B-A12B-v1-Q4_K_M-00001-of-00002.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 40000
--flash-attn on
--temp 0.6
--top-p 0.95
# ==========================================================================
# SKYFALL MODELS
# ==========================================================================
"skyfall-r1-31b-q6k":
name: "Skyfall R1 31B v4 Q6_K_L"
description: "TheDrummer Skyfall R1 31B v4. General-purpose reasoning."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/bartowski_TheDrummer_Skyfall-31B-v4-GGUF/TheDrummer_Skyfall-31B-v4-Q6_K_L.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 40000
--flash-attn on
"skyfall-r1-31b-v4a":
name: "Skyfall R1 31B v4a Q6_K (RP)"
description: "BeaverAI Skyfall R1 v4a variant. RP/creative-focused."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/RP/BeaverAI_Skyfall-R1-31B-v4a-GGUF/Skyfall-R1-31B-v4a-Q6_K.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 40000
--flash-attn on
# ==========================================================================
# CODER MODELS
# ==========================================================================
"qwen3-coder-next":
name: "Qwen3 Coder Next UD-Q4_K_XL"
description: "Latest Qwen3 Coder. Non-reasoning model, optimized for code gen. KB: temp 1.0, top-k 40, min-p 0.01."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/unsloth_Qwen3-Coder-Next-GGUF/Qwen3-Coder-Next-UD-Q4_K_XL.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 32768
--flash-attn on
--temp 1.0
--top-p 0.95
--top-k 40
--min-p 0.01
--repeat-penalty 1.0
# ==========================================================================
# LARGE / SPECIAL-PURPOSE MODELS
# ==========================================================================
"kimik2-q2kxl":
name: "Kimi K2 Instruct UD-Q2_K_XL"
description: "Moonshot Kimi K2. Huge MoE model (8-shard Q2). Limited GPU layers due to size."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/unsloth_Kimi-K2-Instruct-0905-GGUF/UD-Q2_K_XL/Kimi-K2-Instruct-0905-UD-Q2_K_XL-00001-of-00008.gguf
--port ${PORT}
--n-gpu-layers 2
--temp 0.6
--top-p 0.95
# ==========================================================================
# GRANITE MODELS (IBM)
# ==========================================================================
"granite-4-small":
name: "Granite 4.0 Small Q4_K_M"
description: "IBM Granite 4.0 Small. Deterministic utility model for structured tasks."
ttl: 0
cmd: |
/app/llama-server
--context-shift
--model /models/unsloth_granite-4.0-h-small-GGUF/granite-4.0-h-small-Q4_K_M.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 120000
--flash-attn on
--top-p 1.0
--temp 0.0
--top-k 0
"granite-4-micro":
name: "Granite 4.0 Micro Q4_K_M"
description: "IBM Granite 4.0 Micro. Ultra-lightweight for fast structured responses."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--model /models/ibm-granite_granite-4.0-micro-GGUF/granite-4.0-micro-Q4_K_M.gguf
--port ${PORT}
--n-gpu-layers 999
--ctx-size 32768
--flash-attn on
--temp 0.0
--top-p 1.0
# ==========================================================================
# EMBEDDING MODELS (persistent, always loaded)
# ==========================================================================
"embeddinggemma-300M":
name: "Embedding Gemma 300M"
description: "Google Embedding Gemma for vectorization."
ttl: 0
cmd: |
/app/llama-server
--embedding
--pooling cls
--model /models/ggml-org_embeddinggemma-300M-GGUF/embeddinggemma-300M-Q8_0.gguf
--port ${PORT}
--n-gpu-layers 0
--ctx-size 2048
--batch-size 1024
--no-mmap
--ubatch-size 1024
--cont-batching
--threads 24
"qwen3-embedding-0.6B":
name: "Qwen3 Embedding 0.6B"
description: "Qwen3 Embedding model for vectorization. 32K context, last-token pooling (decoder/causal LM)."
ttl: 0
cmd: |
/app/llama-server
--embeddings
--pooling last
--model /models/Qwen_Qwen3-Embedding-0.6B-GGUF/Qwen3-Embedding-0.6B-Q8_0.gguf
--port ${PORT}
--n-gpu-layers 0
--ctx-size 8192
--batch-size 8192
--ubatch-size 2048
--no-mmap
--cont-batching
--threads 24
# ==========================================================================
# RERANKING MODELS (persistent, always loaded)
# ==========================================================================
"qwen3-reranker-0.6B":
name: "Qwen3 Reranker 0.6B"
description: "Qwen3 Reranker for retrieval reranking. Causal LM scoring yes/no logits at last token. Replaces BGE v2."
ttl: 0
cmd: |
/app/llama-server
--reranking
--pooling rank
--model /models/ggml-org_Qwen3-Reranker-0.6B-Q8_0-GGUF/qwen3-reranker-0.6b-q8_0.gguf
--port ${PORT}
--n-gpu-layers 0
--ctx-size 8192
--batch-size 8192
--ubatch-size 2048
--cont-batching
--threads 24
# NOTE: jina-reranker-v3 removed 2025-07-22 — unused, out of rotation.
# Qwen3 Reranker handles all reranking duties.
# ============================================================================
# GROUPS
# - swap: false = models in group can coexist in memory
# - exclusive: false = group can share memory with other groups
# - persistent: true = models never unload (for utility/embedding)
# ============================================================================
groups:
"high-reasoning":
swap: false
exclusive: false
members:
- "qwen3.5-35-a3b"
- "gemma4-31b-dense"
- "nemotron-3-nano-30b"
"heavy-moe":
swap: false
exclusive: false
members:
- "qwen3.5-122b-a10b"
- "nemotron-3-super-120b"
- "kimik2-q2kxl"
- "glm-steam-106b"
"utility":
swap: false
exclusive: false
persistent: true
members:
- "embeddinggemma-300M"
- "qwen3-embedding-0.6B"
- "qwen3-reranker-0.6B"
stacks/rest-server-ana/.env.example
+41
View File
@@ -0,0 +1,41 @@
# rest-server-ana stack tunables. Copy to `.env` on ana-docker.
#
# cp .env.example .env
# # edit if needed
# docker compose up -d
#
# Matches stacks/rest-server-nh3/.env.example — keep them aligned so
# restic clients see the same URL shape against either endpoint.
REST_SERVER_VERSION=latest
# Host port the container listens on (container internal is 8000)
REST_PORT=8000
# Where restic pack files live. TrueNAS NFS share is already mounted at
# /mnt/backup on ana-docker; the "repo/ana" subdir is the historical
# location used by the prior (non-private-repos) rest-server.
#
# With --private-repos, layout becomes:
# ${DATA_DIR}/ana-docker/
# ${DATA_DIR}/ana-ml2/
# ${DATA_DIR}/nh3-docker/
# ${DATA_DIR}/esh-docker-vm/
# Plus the auth file at ${DATA_DIR}/.htpasswd.
DATA_DIR=/mnt/backup/restic/repo/ana
# UID/GID the container process runs as. Must match the owner of
# DATA_DIR so NFS root_squash doesn't bite. On ana-docker this is the
# `lkraven` user (1000:1000).
REST_UID=1000
REST_GID=1000
# Timezone — affects log lines and /metrics timestamps
TZ=America/Los_Angeles
# Extra rest-server flags. Examples:
# --prometheus-no-auth — make /metrics public (needed if Beszel or
# Prometheus scrapes without creds)
# --no-verify-upload — trust the client's hash; faster writes
# Leave blank unless you have a reason.
EXTRA_OPTIONS=
stacks/rest-server-ana/README.md
+143
View File
@@ -0,0 +1,143 @@
# rest-server-ana
Anaheim-site restic backup endpoint. Replaces the older `restic` stack on ana-docker with the same auth model as `rest-server-nh3` on the Synology, so every client host uses identical URL shapes against either endpoint.
**Server:** ana-docker (`10.250.50.70`)
**Port:** `http://10.250.50.70:8000`
**Data:** `/mnt/backup/restic/repo/ana/` (TrueNAS NFS mount on the host)
Paired with:
- **`rest-server-nh3`** on the Synology (`10.100.50.50:8000`, data on Btrfs).
- A cross-site rsync job (TBD, on ana-docker) that mirrors each site's data tree to the other so either NAS can fully restore either site's hosts.
## What changed from the old `restic` stack
| | old `restic` on ana-docker | this stack |
|---|---|---|
| `--private-repos` | no | **yes** |
| `--append-only` | no | **yes** |
| `--prometheus` | no | **yes** |
| healthcheck | no | yes |
| `.env`-driven | no | yes |
| restart policy | none | `unless-stopped` |
| image version | floating `latest` | `${REST_SERVER_VERSION}` |
| stack dir on server | `/opt/docker/compose/restic/` | `/opt/docker/compose/rest-server-ana/` |
Data path is unchanged (`/mnt/backup/restic/repo/ana/`) so nothing new needs to be allocated on TrueNAS.
## Pre-deploy: clean the data dir and create htpasswd
Since there's nothing in the existing path we want to keep, start fresh so the on-disk layout matches `--private-repos`:
```bash
ssh ana-docker '
# Stop the old stack so port 8000 and the data dir are free
cd /opt/docker/compose/restic
docker compose down
# Wipe the old non-private-repos layout
sudo rm -rf /mnt/backup/restic/repo/ana/*
sudo rm -rf /mnt/backup/restic/repo/ana/.htpasswd # if present
# Create the htpasswd file. Use the same passwords here as on the NH3
# Synology so each host has one credential that works at either endpoint.
sudo touch /mnt/backup/restic/repo/ana/.htpasswd
sudo chmod 600 /mnt/backup/restic/repo/ana/.htpasswd
'
# Generate htpasswd entries locally (one per host) and append. Using the
# `httpd:2.4-alpine` throwaway container so we do not depend on
# apache2-utils being installed on ana-docker.
for user in ana-docker ana-ml2 nh3-docker esh-docker-vm; do
read -rs -p "password for $user (must match the NH3 Synology): " pw; echo
docker run --rm httpd:2.4-alpine htpasswd -nbB "$user" "$pw" \
| ssh ana-docker 'sudo tee -a /mnt/backup/restic/repo/ana/.htpasswd >/dev/null'
done
```
If you run that locally and don't have Docker here, equivalent on the server:
```bash
ssh ana-docker "docker run --rm httpd:2.4-alpine htpasswd -nbB <user> '<pw>'" \
| ssh ana-docker 'sudo tee -a /mnt/backup/restic/repo/ana/.htpasswd >/dev/null'
```
## Deploy
Stage the new stack and push it:
```bash
# Stage the stack into the mirror (if not already done via sync-stacks.sh)
mkdir -p stacks-mirror/ana-docker/rest-server-ana
cp stacks/rest-server-ana/compose.yaml stacks/rest-server-ana/.env.example \
stacks-mirror/ana-docker/rest-server-ana/
scripts/deploy-stack.sh ana-docker rest-server-ana
```
Confirm at the prompt. Then on the server:
```bash
ssh ana-docker '
cd /opt/docker/compose/rest-server-ana
cp -n .env.example .env
docker compose config
docker compose up -d
docker compose logs --tail=30
'
```
## Retire the old stack
Once the new one is healthy and the first repo has initialized successfully from a client:
```bash
ssh ana-docker '
cd /opt/docker/compose/restic
docker compose down
# Optionally remove the old stack dir (keep it for a release or two
# in case you need to roll back):
# rm -rf /opt/docker/compose/restic
'
```
## Verify
```bash
# 401 from the root — service up, auth enforced
curl -sS -o /dev/null -w 'unauth status=%{http_code}\n' \
http://10.250.50.70:8000/
# 200 / 404 from a real user+password — auth valid, --private-repos path OK
curl -sS -o /dev/null -w 'auth status=%{http_code}\n' \
-u ana-docker:<password> http://10.250.50.70:8000/ana-docker/
# Init a repo from a client host (one-time per host)
ssh ana-docker '
export RESTIC_REPOSITORY="rest:http://ana-docker:<rest-pw>@10.250.50.70:8000/ana-docker/"
export RESTIC_PASSWORD="<client-side-encryption-passphrase>"
restic init
'
```
## Prune ceremony
Same as `rest-server-nh3` — prune is blocked by `--append-only`. Two options, pick one per endpoint:
- **Temporary flag flip:** edit compose, remove `--append-only` from `OPTIONS`, `docker compose up -d`, run `restic forget --prune` from origin hosts, put the flag back, `docker compose up -d`. Quarterly change.
- **Second endpoint on a different port:** stand up a sibling container (e.g. port `8001`) against the same data dir without `--append-only`, reachable only from a trusted host. Everyday backups still hit `:8000`.
If you go the second-endpoint route, copy this stack to `stacks/rest-server-ana-prune/` with `REST_PORT=8001` and `--append-only` removed from the compose.
## Off-site replication
Scheduled on ana-docker (to be written):
```bash
# Pull NH3's tree down to this side
rsync -avz --delete admin@10.100.50.50:/volume1/Backup/restic/ /mnt/backup/restic-mirror-nh3/
# Push our tree to NH3
rsync -avz --delete /mnt/backup/restic/repo/ana/ admin@10.100.50.50:/volume1/Backup/restic-mirror-ana/
```
Two unidirectional syncs, each running in the direction its data flows. Prune runs only at the origin so the mirror shrinks correctly.
stacks/rest-server-ana/compose.yaml
+59
View File
@@ -0,0 +1,59 @@
# rest-server (Anaheim) — restic backup target for the fleet.
#
# Deploys to ana-docker. Data dir is on the TrueNAS NFS mount
# (/mnt/backup/restic/repo/ana) so snapshots on the NAS side protect the
# backup blobs themselves.
#
# Mirrors stacks/rest-server-nh3/ in every meaningful way — same auth
# model, same on-disk layout, same operational semantics — so each client
# host uses an identical URL shape against either endpoint:
#
# rest:http://<user>:<pw>@10.100.50.50:8000/<user>/ (NH3 Synology)
# rest:http://<user>:<pw>@10.250.50.70:8000/<user>/ (this stack)
#
# Auth model:
# --private-repos : URL path must start with /<user>/ and the HTTP
# basic-auth user must match. Per-host repos are
# strictly isolated.
# --append-only : on-disk data can be added but not removed or
# rewritten; a compromised host can't wipe its own
# history. Prune requires disabling this (see README).
#
# Credentials come from /data/.htpasswd — see README for populating it.
#
# All tunables live in .env — edit that, not this file.
services:
rest-server:
image: restic/rest-server:${REST_SERVER_VERSION}
container_name: rest-server
restart: unless-stopped
# Run as the UID that owns the NFS-backed data dir, so file I/O
# is not subject to NFS root_squash. On ana-docker this is lkraven (1000).
user: "${REST_UID:-1000}:${REST_GID:-1000}"
ports:
- "${REST_PORT}:8000"
volumes:
- ${DATA_DIR}:/data
environment:
- OPTIONS=--private-repos --append-only --prometheus ${EXTRA_OPTIONS:-}
- TZ=${TZ:-America/Los_Angeles}
healthcheck:
test: ["CMD-SHELL", "wget -qO- http://localhost:8000/metrics >/dev/null 2>&1 || [ $? -eq 6 ] && exit 0 || exit 1"]
interval: 30s
timeout: 10s
retries: 3
start_period: 15s
networks:
- tnet
labels:
- homepage.group=PFI-ANA
- homepage.name=Restic (rest-server)
- homepage.icon=mdi-cloud-upload
- homepage.description=Anaheim restic endpoint (data on TrueNAS NFS)
- homepage.href=http://10.250.50.70:${REST_PORT}
networks:
tnet:
name: traefik-net
external: true
stacks/rest-server-nh3/.env.example
+35
View File
@@ -0,0 +1,35 @@
# rest-server (NH3 Synology) tunables. Copy to `.env` on the Synology.
#
# cp .env.example .env
# # edit if needed
# docker compose up -d
#
# Synology-specific notes:
# - Put this stack under /volume1/docker/compose/rest-server/
# (Container Manager's default project root pattern on DSM 7.x).
# - Point DATA_DIR at a Btrfs share you own — /volume1/Backup/restic
# is the natural choice since /volume1/Backup is already the
# fleet-facing backup share (exported via NFS).
REST_SERVER_VERSION=latest
# Host port the container listens on (container internal port is 8000)
REST_PORT=8000
# Where restic pack files live on the Synology. Must be a writable
# Btrfs path. Will hold one subdir per user (--private-repos layout):
# ${DATA_DIR}/ana-docker/
# ${DATA_DIR}/ana-ml2/
# ${DATA_DIR}/nh3-docker/
# ${DATA_DIR}/esh-docker-vm/
# Plus the auth file at ${DATA_DIR}/.htpasswd.
DATA_DIR=/volume1/Backup/restic
# Timezone — affects log lines and the /metrics timestamps
TZ=America/Los_Angeles
# Any extra rest-server flags (rare). Some useful ones:
# --no-verify-upload — trust the client's hash; faster writes
# --max-size=<bytes> — cap per-repo size
# Leave blank unless you have a reason.
EXTRA_OPTIONS=
stacks/rest-server-nh3/README.md
+124
View File
@@ -0,0 +1,124 @@
# rest-server-nh3
NH3-site restic backup endpoint. Runs in Synology Container Manager on `10.100.50.50` and stores pack files on a Btrfs share so Synology snapshots protect against local corruption.
**Server:** Synology RS2418+ at `10.100.50.50`
**Port:** `http://10.100.50.50:8000` (configurable via `.env`)
**Data:** `/volume1/Backup/restic/` (configurable)
Paired with the existing `restic rest-server` on **ana-docker** (`http://10.250.50.70:8000`, data on TrueNAS NFS) as the Anaheim-side endpoint. Each fleet host backs up to the rest-server closest to it; an rsync job on ana-docker mirrors the two trees against each other for off-site redundancy.
## Auth model
`--private-repos` + `--append-only`, enforced via htpasswd:
- One HTTP basic-auth user **per host** (`ana-docker`, `ana-ml2`, `nh3-docker`, `esh-docker-vm`).
- Each user can only write under `/<username>/…` — a compromised host can't see or delete another host's data.
- Append-only means a compromised client can add to its own repo but can't rewrite or delete existing packs, so ransomware on a backed-up host doesn't destroy history.
- **Trade-off:** `restic forget --prune` can't run against an append-only endpoint. Prune ceremony described at the bottom of this file.
## Pre-deploy: create the data path and htpasswd
On the Synology (SSH in as an admin-capable user, or DSM *File Station*):
```bash
# 1. Create the restic data share on a Btrfs volume
ssh admin@10.100.50.50 'sudo mkdir -p /volume1/Backup/restic && \
sudo chown 1000:1000 /volume1/Backup/restic && \
sudo chmod 700 /volume1/Backup/restic'
# 2. Generate htpasswd entries. The Synology doesn't ship apache2-utils,
# so use a throwaway container:
ssh admin@10.100.50.50 'cd /volume1/Backup/restic && \
sudo touch .htpasswd && sudo chown 1000:1000 .htpasswd && sudo chmod 600 .htpasswd'
for user in ana-docker ana-ml2 nh3-docker esh-docker-vm; do
read -s -p "password for $user: " pw; echo
ssh admin@10.100.50.50 \
"docker run --rm httpd:2.4-alpine htpasswd -nbB $user '$pw'" \
| ssh admin@10.100.50.50 "sudo tee -a /volume1/Backup/restic/.htpasswd >/dev/null"
done
```
Record every password in your off-host password manager (1Password / Vaultwarden etc.) — you'll paste them into Backrest and into the systemd timer configs later.
## Deploy
In Synology **Container Manager**:
1. *Project* → **Create** → Name `rest-server`, Path `/volume1/docker/compose/rest-server/`.
2. Copy `compose.yaml` into the project path; copy `.env.example` → `.env` and edit if needed (default `REST_PORT=8000` and `DATA_DIR=/volume1/Backup/restic` should be fine).
3. Start the project.
CLI equivalent (if you have SSH + a shell account that can run Docker on the NAS):
```bash
ssh admin@10.100.50.50
sudo mkdir -p /volume1/docker/compose/rest-server
sudo chown $USER /volume1/docker/compose/rest-server
cd /volume1/docker/compose/rest-server
# scp the files from this workspace, then:
cp .env.example .env
docker compose config
docker compose up -d
docker compose logs -f
```
## Verify
From this workstation:
```bash
# Should return "200 OK" or redirect to /metrics; anything 5xx is a problem.
curl -u ana-docker:<password> -sv http://10.100.50.50:8000/ana-docker/ -o /dev/null
# Once restic is wired up, init the repo (one-time, per host):
RESTIC_REPOSITORY='rest:http://ana-docker:<password>@10.100.50.50:8000/ana-docker/' \
RESTIC_PASSWORD='<client-side-encryption-passphrase>' \
restic init
```
Restic URI shape for each host (paste into Backrest when adding the repo):
```
rest:http://<user>:<pass>@10.100.50.50:8000/<user>/
```
## Prune ceremony (because of --append-only)
Because `--append-only` blocks deletes, `restic forget --prune` will fail against the live endpoint. Two options, pick one and stick with it:
### Option A — temporary flag flip (simplest, requires a maintenance window)
1. On the Synology, edit the stack's `.env` and set `EXTRA_OPTIONS=--no-auth` — **only kidding, don't.** Set `EXTRA_OPTIONS= ` and comment out `--append-only` in the compose `OPTIONS=` line (or parameterize if you prefer).
2. `docker compose up -d` to restart with deletes allowed.
3. Run `restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-yearly 3 --prune` from the origin host.
4. Restore `--append-only` and `docker compose up -d`.
Treat this as a quarterly change, not a cron job. Schedule it so you're present if restic hits anything weird.
### Option B — second endpoint on a different port (automation-friendly)
Stand up a second `rest-server` container against the same `DATA_DIR` without `--append-only`, listening on e.g. `8001`, reachable only from within the Synology / over VPN. A scheduled prune job hits that endpoint; day-to-day backup traffic continues to hit `:8000` in append-only mode.
If you end up wanting this, copy this stack to `stacks/rest-server-nh3-prune/` with `REST_PORT=8001` and `--append-only` removed.
## Off-site replication
Scheduled on ana-docker:
```bash
# Example — not the final script, just illustrating the shape.
rsync -avz --delete \
admin@10.100.50.50:/volume1/Backup/restic/ \
/mnt/backup/restic-mirror-nh3/
```
`rsync` is safe because restic packs are immutable once written — nothing under `/volume1/Backup/restic/<user>/data/` gets rewritten, only added or (during prune) removed. A raw `rsync --delete` with prune running only on the origin side is enough; no filesystem-level locks required.
## What doesn't live here
- No `.env` in the committed copy — only `.env.example`.
- `.htpasswd` is never checked in, never synced via `sync-stacks.sh` (its `*.ht*` isn't in the global exclude but the data dir is outside the stack path).
- Client-side restic passwords (the encryption passphrase for each repo) are separate from the HTTP auth passwords and never stored on the Synology.
stacks/rest-server-nh3/compose.yaml
+40
View File
@@ -0,0 +1,40 @@
# rest-server (NH3 Synology) — restic backup target for the fleet.
#
# Deploys into Synology Container Manager on 10.100.50.50. Data lives on
# a Btrfs shared folder so it gets Synology snapshots + optional
# replication to a sibling share if you configure one later.
#
# Auth model:
# --private-repos : every URL path must start with /<username>/ and the
# HTTP basic-auth user must match. One user per host.
# Per-host repos are strictly isolated.
# --append-only : on-disk data can be ADDED but not REMOVED or REWRITTEN.
# A compromised host can't delete its own history.
# Prune requires disabling this (see README).
#
# Credentials come from /data/.htpasswd — see README for how to populate
# it. That file is mounted read-only into the container.
#
# All tunables live in .env — edit that, not this file.
services:
rest-server:
image: restic/rest-server:${REST_SERVER_VERSION}
container_name: rest-server
restart: unless-stopped
ports:
- "${REST_PORT}:8000"
volumes:
- ${DATA_DIR}:/data
environment:
- OPTIONS=--private-repos --append-only --prometheus ${EXTRA_OPTIONS:-}
- TZ=${TZ:-America/Los_Angeles}
# rest-server stores repos under /data and looks for /data/.htpasswd
# automatically — no extra bind mount needed as long as the htpasswd
# file is created inside DATA_DIR before startup.
healthcheck:
test: ["CMD-SHELL", "wget -qO- http://localhost:8000/metrics >/dev/null || exit 1"]
interval: 30s
timeout: 10s
retries: 3
start_period: 15s
stacks/vllm-qwen3/.env.example
+40
View File
@@ -0,0 +1,40 @@
# vllm-qwen3 stack tunables. Copy this to `.env` on the server before deploying.
#
# cp .env.example .env
# # edit .env with real values
# docker compose up -d
# Image version — pin for reproducibility (`latest` for edge)
VLLM_VERSION=latest
# Host ports (container always listens on 8000 internally)
EMBED_PORT=8001
RERANK_PORT=8002
# GPU assignment — both services share this GPU
# (ana-ml2 has 0 and 1; default 1 keeps 0 free for heavy LLM work)
GPU_ID=1
# Models — reference by full repo name in API requests
EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B
RERANK_MODEL=Qwen/Qwen3-Reranker-0.6B
# GPU memory split — fractions are of TOTAL GPU memory, not free memory.
# When two vLLM services share a GPU, each profiler needs its own slice to
# fit both the model and KV cache, so small values cause the second-to-start
# service to OOM on KV cache allocation. 0.40 + 0.40 leaves ~20% headroom
# and is comfortably above the minimum for two 0.6B Qwen3 models at 8k ctx.
EMBED_GPU_MEM_UTIL=0.40
RERANK_GPU_MEM_UTIL=0.40
# Context length caps — lower these if VRAM is tight.
# Qwen3-Embedding supports up to 32k; reranker up to 32k.
EMBED_MAX_MODEL_LEN=8192
RERANK_MAX_MODEL_LEN=8192
# Optional API key — leave blank for no auth (fine on the internal network).
# If set, both services require `Authorization: Bearer <key>`.
API_KEY=
# HuggingFace token — only needed for gated models
HF_TOKEN=
stacks/vllm-qwen3/README.md
+80
View File
@@ -0,0 +1,80 @@
# vllm-qwen3
Qwen3 embedding + reranker served via vLLM. Replaces the unmaintained Infinity stack.
**Server:** ana-ml2
**Ports:** `8001` (embed), `8002` (rerank) — both configurable via `.env`
**GPU:** both services share GPU 1 by default (configurable)
## Why two services
vLLM runs **one model per process**, so embedding and reranking each get their own container. Both pin to the same GPU and split VRAM via `--gpu-memory-utilization`. Both use `--runner pooling` so the OpenAI server exposes `/v1/embeddings` (for the embedder) and `/rerank`, `/score` (for the reranker, which also needs the `--hf-overrides` described below).
## Reranker caveat
Qwen/Qwen3-Reranker-0.6B is a causal-LM checkpoint. The `--hf-overrides` flag in `compose.yaml` re-maps it to `Qwen3ForSequenceClassification` so vLLM's `/rerank` and `/score` endpoints work and the model emits only `no`/`yes` class logits instead of the full 151k-token distribution.
If that override breaks after a vLLM upgrade, the pre-converted checkpoint `tomaarsen/Qwen3-Reranker-0.6B-seq-cls` is a drop-in replacement that needs no overrides — set `RERANK_MODEL=tomaarsen/Qwen3-Reranker-0.6B-seq-cls` in `.env` and remove the `--hf-overrides` line from the compose.
## Deploy
```bash
# On ana-ml2:
sudo mkdir -p /opt/docker/compose/vllm-qwen3
sudo chown $USER /opt/docker/compose/vllm-qwen3
cd /opt/docker/compose/vllm-qwen3
# Copy compose.yaml + .env.example here (e.g. via scp from this workspace)
cp .env.example .env
# edit .env — pick GPU, ports, memory split, etc.
# Pre-download models (optional, speeds first boot)
HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^EMBED_MODEL .env | cut -d= -f2)"
HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^RERANK_MODEL .env | cut -d= -f2)"
# Dry-parse
docker compose config
# Launch
docker compose up -d
docker compose logs -f
```
First boot compiles CUDA graphs and can take 2–3 minutes per service. The `start_period: 180s` healthcheck grace reflects that.
## Verify
```bash
# Health
curl -s http://localhost:8001/health
curl -s http://localhost:8002/health
# Embedding (OpenAI-compatible)
curl -s http://localhost:8001/v1/embeddings \
-H "Content-Type: application/json" \
-d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":["hello world"]}' | jq .
# Reranker
curl -s http://localhost:8002/rerank \
-H "Content-Type: application/json" \
-d '{"model":"Qwen/Qwen3-Reranker-0.6B","query":"what is a cat","documents":["cats are mammals","dogs bark"]}' | jq .
# Listed models
curl -s http://localhost:8001/v1/models | jq .
curl -s http://localhost:8002/v1/models | jq .
```
## Scaling knobs
- **`EMBED_GPU_MEM_UTIL` / `RERANK_GPU_MEM_UTIL`** — fractions of **total** GPU VRAM each service reserves (not of free VRAM). Both services profile independently, so each slice must be large enough to fit that service's model + KV cache with no knowledge of the other. Setting them too low causes the second-to-start container to OOM on KV cache allocation with `Available KV cache memory: -X.XX GiB`. Default 0.40/0.40 (= 0.80 total) leaves ~20% GPU headroom and works cleanly for the 0.6B pair at 8k context; raise for 4B/8B variants or drop `max-model-len` if you need more room.
- **`EMBED_MAX_MODEL_LEN` / `RERANK_MAX_MODEL_LEN`** — lower to reduce KV-cache allocation if VRAM is tight. Qwen3 supports up to 32k natively.
- **Larger models** — Qwen3-Embedding/Reranker come in 0.6B / 4B / 8B. Swap `EMBED_MODEL` / `RERANK_MODEL` and bump the memory fractions accordingly.
- **Separate GPUs** — if contention hurts latency, split them: add a second `GPU_ID_RERANK` variable and point each service at its own device. (Requires a small compose edit; currently both share `${GPU_ID}`.)
## Migrating off Infinity
Once this stack is verified stable:
1. Stop the infinity stack (`docker compose down` under `/opt/docker/compose/infinity/`).
2. Update consumers (AIPA agents, LibreChat RAG) to point at `:8001` for embeddings and `:8002` for rerank.
3. Delete `stacks/infinity/` from this workspace.
stacks/vllm-qwen3/compose.yaml
+133
View File
@@ -0,0 +1,133 @@
# vLLM — Qwen3 Embedding + Reranker (one stack, two services).
#
# Replaces the unmaintained Infinity stack. vLLM runs one model per process,
# so this stack brings up two containers sharing a single GPU:
#
# vllm-embed — Qwen3-Embedding served as an OpenAI /v1/embeddings server
# vllm-rerank — Qwen3-Reranker served as a /rerank + /score server
#
# The reranker is a causal-LM checkpoint; --hf-overrides re-maps it to
# Qwen3ForSequenceClassification so vLLM's reranking endpoints work and the
# model only emits two class logits (no/yes) instead of the full 151k vocab.
#
# All tunables live in .env — edit that, not this file.
#
# Pre-download models to avoid first-run delay:
# HF_HOME=/tank/aimodels/huggingface hf download Qwen/Qwen3-Embedding-0.6B
# HF_HOME=/tank/aimodels/huggingface hf download Qwen/Qwen3-Reranker-0.6B
services:
vllm-embed:
image: vllm/vllm-openai:${VLLM_VERSION}
container_name: vllm-embed
restart: unless-stopped
ipc: host
ports:
- "${EMBED_PORT}:8000"
volumes:
- /tank/aimodels/huggingface:/hfcache
environment:
- HF_HOME=/hfcache
- HF_HUB_CACHE=/hfcache/hub
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
- VLLM_API_KEY=${API_KEY:-}
command:
- ${EMBED_MODEL}
- --served-model-name
- ${EMBED_MODEL}
- --runner
- pooling
- --host
- 0.0.0.0
- --port
- "8000"
- --gpu-memory-utilization
- ${EMBED_GPU_MEM_UTIL}
- --max-model-len
- ${EMBED_MAX_MODEL_LEN}
- --dtype
- auto
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids:
- "${GPU_ID}"
capabilities:
- gpu
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 180s
networks:
- tnet
labels:
- homepage.group=AI Systems
- homepage.name=vLLM Embed (Qwen3)
- homepage.icon=mdi-vector-arrange-below
- homepage.description=Qwen3 Embedding via vLLM (ana-ml2)
- homepage.href=http://10.250.50.54:${EMBED_PORT}/docs
vllm-rerank:
image: vllm/vllm-openai:${VLLM_VERSION}
container_name: vllm-rerank
restart: unless-stopped
ipc: host
ports:
- "${RERANK_PORT}:8000"
volumes:
- /tank/aimodels/huggingface:/hfcache
environment:
- HF_HOME=/hfcache
- HF_HUB_CACHE=/hfcache/hub
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
- VLLM_API_KEY=${API_KEY:-}
command:
- ${RERANK_MODEL}
- --served-model-name
- ${RERANK_MODEL}
- --runner
- pooling
- --hf-overrides
- '{"architectures":["Qwen3ForSequenceClassification"],"classifier_from_token":["no","yes"],"is_original_qwen3_reranker":true}'
- --host
- 0.0.0.0
- --port
- "8000"
- --gpu-memory-utilization
- ${RERANK_GPU_MEM_UTIL}
- --max-model-len
- ${RERANK_MAX_MODEL_LEN}
- --dtype
- auto
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids:
- "${GPU_ID}"
capabilities:
- gpu
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 180s
networks:
- tnet
labels:
- homepage.group=AI Systems
- homepage.name=vLLM Rerank (Qwen3)
- homepage.icon=mdi-sort-variant
- homepage.description=Qwen3 Reranker via vLLM (ana-ml2)
- homepage.href=http://10.250.50.54:${RERANK_PORT}/docs
networks:
tnet:
name: traefik-net
external: true