commit e376d0aec9386cea9f4beab481199bb7d95df0b1 Author: Vuong Hoang Date: Mon Apr 20 14:23:18 2026 -0700 Initial commit: PFI fleet inventory, stacks, tooling, and backup pipeline Captures the full workspace state built up to this point: - CLAUDE.md + README.md describing conventions and the four-host fleet (ana-ml2, ana-docker, nh3-docker, esh-docker-vm). - Per-host notes under servers// with ssh-target fallback files and latest system-details snapshots (two in-compose credential leaks scrubbed; the upstream compose files still need to move those to .env). - scripts/: server_inspect.sh (read-only remote diagnostic), refresh-server-info.sh (dir-driven discovery + snapshot capture with validation warnings), add-host.sh, sync-stacks.sh (pull compose/conf trees), deploy-stack.sh (push with per-file diff + prompt). - stacks/: canonical compose for backrest, beszel, dozzle, llama-swap, rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired infinity reference. All use the .env-driven + traefik-net + homepage label pattern. - configs/restic/ana-docker/: first resticprofile config + pre-backup hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates for the other three hosts to come. - docs/pfi/: general infrastructure reference carried over. - .gitignore excludes .env, stacks-mirror/, and assorted secret/state filenames to prevent re-leaks on later commits. diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..e82d495 --- /dev/null +++ b/.gitignore @@ -0,0 +1,28 @@ +# Secrets — real .env files must never land here, only .env.example templates. +.env +.env.local +.env.*.local +**/.env +!**/.env.example + +# Live mirror of server /opt/docker/{compose,conf}/ trees pulled by +# sync-stacks.sh. Contains upstream compose files that can carry embedded +# plaintext credentials (e.g. legacy seafile/paperless configs), so we +# don't track them in git. Audited, hand-curated copies live under +# stacks// and are the source of truth. +stacks-mirror/ + +# Staged htpasswd / secrets files that might get written to /tmp during +# helper scripts. +htpasswd-new +*.netrc + +# Editor / OS cruft +.DS_Store +*.swp +*.swo +*~ + +# Logs +*.log +*.log.* diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..be3f934 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,150 @@ +# CLAUDE.md + +This workspace is for managing PFI infrastructure — servers, Docker stacks, and related configs. Spawn a dedicated Claude Code session here when working on infra so it doesn't clutter AIPA-MCP development context. + +## Purpose + +- Inventory of servers and their state +- Canonical copies of Docker Compose stacks deployed on those servers +- Scripts for inspecting and managing the infrastructure +- Conventions so all stacks look the same + +This is a **reference workspace** — the authoritative copies of compose files and configs live **on the servers** under `/opt/docker/compose//` and `/opt/docker/conf//`. This workspace mirrors them for version control, editing, and planning. + +## Conventions (enforce for every new stack) + +Observed and standardized across servers: + +- **Compose location on server:** `/opt/docker/compose//compose.yaml` +- **Config mounts on server:** `/opt/docker/conf//...` +- **Networks:** external `traefik-net`, aliased as `tnet` in compose + ```yaml + networks: + tnet: + name: traefik-net + external: true + ``` +- **GPU reservation:** prefer `deploy.resources.reservations.devices` with explicit `device_ids` for pinning + ```yaml + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: ["1"] + capabilities: [gpu] + ``` +- **Tunables:** `.env` in the same directory as `compose.yaml` — keep the compose file constant, edit the `.env` +- **Named volumes** for service state (pattern: `_`) +- **Bind mounts** only for: model files (`/tank/aimodels/...`), config files (`/opt/docker/conf/...`), docker socket where required +- **Restart policy:** `restart: unless-stopped` for daemons +- **Homepage labels** on user-facing services: + ```yaml + labels: + - homepage.group=AI Systems + - homepage.name= + - homepage.icon=mdi- + - homepage.description= + - homepage.href=http://: + ``` +- **Healthchecks** on services that expose HTTP + +## Servers + +| Name | IP | Site | Role | Details | +|------|-----|------|------|---------| +| ana-ml2 | 10.250.50.54 | Anaheim (`10.250.0.0/16`) | GPU / AI inference | `servers/ana-ml2/README.md` | +| ana-docker | 10.250.50.70 | Anaheim (`10.250.0.0/16`) | General-purpose Docker host (non-GPU) | `servers/ana-docker/README.md` | +| nh3-docker | 10.100.50.40 | NH3 (`10.100.0.0/16`) | General-purpose Docker host (non-GPU) | `servers/nh3-docker/README.md` | +| esh-docker-vm | 10.0.50.45 | ESH home lab (`esteban.net`, `10.0.50.0/24`) | Home-lab Docker host (non-PFI scope) | `servers/esh-docker-vm/README.md` | + +**Placement rules:** +- GPU-required stacks → `ana-ml2`. +- Anaheim non-GPU services → `ana-docker`. +- NH-site non-GPU services → `nh3-docker`. +- ESH home-lab workloads (`esteban.net`) → `esh-docker-vm`. Not part of the PFI colo topology, but shares monitoring/backup tooling. +- Cross-site services (e.g. Beszel hub, Dozzle hub) live on `ana-docker` and pull from agents on the other hosts. + +## How to refresh a server's state + +```bash +# Show help (no args) +scripts/refresh-server-info.sh + +# Refresh every host discovered under servers/*/ +scripts/refresh-server-info.sh all + +# Refresh a specific host (must match a servers// dir; ssh_config +# entry or servers//ssh-target handles how to reach it) +scripts/refresh-server-info.sh ana-docker +``` + +Fleet-wide runs require the literal `all` keyword — no-args prints help so you can't accidentally hit every host by forgetting a name. + +The script pipes `server_inspect.sh` over SSH via stdin (no scp, no remote cleanup) and writes each `servers//system-details.txt` atomically — a failed run never clobbers the previous snapshot. The inspect script itself is read-only. + +Each server dir can hold an `ssh-target` file (one line, `` or `@`) as a fallback for when the dir name doesn't resolve via DNS or `~/.ssh/config`. The script prefers whatever ssh would resolve normally and only consults the file when that fails. + +To register a new server: + +```bash +scripts/add-host.sh +scripts/refresh-server-info.sh # pull the first snapshot +``` + +To audit discovery without touching the network (checks permissions, unresolvable names with no fallback, missing README / system-details, malformed `ssh-target`): + +```bash +scripts/refresh-server-info.sh --validate-only all +scripts/refresh-server-info.sh --validate-only +``` + +## Stack mirror (pull / push) + +Compose and config trees are mirrored into `stacks-mirror///` so they can be diffed and version-controlled. Pull is fleet-wide and safe; push is one stack at a time with a diff + prompt. + +```bash +# Pull compose + conf from every host into stacks-mirror/ +scripts/sync-stacks.sh +scripts/sync-stacks.sh --dry-run # see what would change +scripts/sync-stacks.sh ana-docker # one host + +# Push a local stack back to the server (diffs each file, prompts y/N) +scripts/deploy-stack.sh +scripts/deploy-stack.sh --compose # skip conf +scripts/deploy-stack.sh --conf # skip compose +``` + +**Opt-out per stack:** create `stacks-mirror///.no-sync` (skip both sides) or `stacks-mirror///conf/.no-sync` (skip conf only). + +**Always excluded in both directions** (secrets / runtime state): `.env`, `.env.*`, `acme.json`, `client_secrets.json`, `*.pem`, `*.key`, `*.crt`, `*.pfx`, `*.sqlite`, `*.sqlite3`, `*.db`, `*.log`, `*.log.*`, `*.pid`, `hub/`, `logs/`. + +Requires `rsync` installed on this workstation and every host you sync against (`apt install rsync`). + +## Layout + +``` +eshpfi-management/ +├── CLAUDE.md # this file +├── README.md # human-facing overview +├── scripts/ +│ └── server_inspect.sh # gather server state for compose planning +├── servers/ +│ └── / +│ ├── README.md +│ └── system-details.txt # latest server_inspect output +├── stacks/ +│ └── / +│ ├── compose.yaml # deployed to /opt/docker/compose// +│ ├── .env.example # template; real .env lives on server +│ └── README.md # what this stack does, how to deploy +└── docs/ + └── pfi/ # general PFI infrastructure reference +``` + +## Working rules + +- **Copies, not symlinks.** Files here reflect what's on the server at the time of the last sync. When you edit here, the server doesn't change until you deploy. +- **Never commit secrets.** Use `.env.example` templates; real `.env` files (with tokens, passwords) live on the server and are gitignored if/when this becomes a git repo. +- **Surgical edits.** When fixing one stack, don't touch unrelated ones. Follow AIPA-MCP's CLAUDE.md rules about scope discipline. +- **Sanity-check before deploying.** Run `docker compose config` (dry parse) before `docker compose up -d` on the server. diff --git a/README.md b/README.md new file mode 100644 index 0000000..8b461f9 --- /dev/null +++ b/README.md @@ -0,0 +1,117 @@ +# eshpfi-management + +Infrastructure management workspace for the PFI fleet (plus the ESH home-lab host). Tracks server state, canonical Docker Compose stacks, per-host configs, and the tooling that moves them around. + +See **[CLAUDE.md](CLAUDE.md)** for the full set of conventions and the rules Claude Code sessions follow when working here. + +## The fleet + +| Host | IP | Site | Role | +|---|---|---|---| +| ana-ml2 | `10.250.50.54` | Anaheim (`10.250.0.0/16`) | GPU / AI inference | +| ana-docker | `10.250.50.70` | Anaheim | General-purpose Docker + cross-site hubs | +| nh3-docker | `10.100.50.40` | NH3 (`10.100.0.0/16`) | General-purpose Docker (NH site) | +| esh-docker-vm | `10.0.50.45` | ESH home lab (`esteban.net`) | Home-lab Docker (non-PFI scope) | + +Per-host snapshots of the running system live under `servers//system-details.txt`, refreshed via `scripts/refresh-server-info.sh`. + +## Layout + +``` +. +├── CLAUDE.md # conventions; loaded by Claude Code sessions +├── README.md # this file +├── scripts/ # workstation tooling +│ ├── server_inspect.sh # read-only diagnostic, runs on remote via stdin +│ ├── refresh-server-info.sh # pull fresh system-details.txt for one/all hosts +│ ├── add-host.sh # register a new server (writes servers//ssh-target) +│ ├── sync-stacks.sh # pull /opt/docker/{compose,conf}/ → stacks-mirror/ +│ └── deploy-stack.sh # push stacks-mirror/// with diff + prompt +├── servers/ # per-host notes + latest snapshot + ssh-target fallback +│ └── / +│ ├── README.md +│ ├── system-details.txt # regenerate on demand +│ └── ssh-target # or @, used when DNS fails +├── stacks/ # canonical compose files (source of truth) +│ └── / +│ ├── compose.yaml +│ ├── .env.example +│ └── README.md +├── stacks-mirror/ # gitignored — live mirror from sync-stacks.sh +├── configs/ # host-level config files that aren't docker-compose +│ └── restic// # resticprofile configs + pre-backup hooks +└── docs/ # general reference (network, models, proxmox, etc.) + └── pfi/ +``` + +## Current stacks + +**GPU (ana-ml2):** +- `llama-swap` — GGUF model swapper via llama.cpp (port 9292) +- `vllm-qwen3` — embeddings (8001) + reranker (8002) via vLLM + +**Anaheim non-GPU (ana-docker):** +- `traefik`, `crowdsec`, `gitea`, `vaultwarden`, `synapse`, `seafile`, `searxng`, `openwebui`, `sillytavern`, `mailrise`, `rustdesk`, `dockge`, `it-tools` +- Fleet services: `beszel` (metrics hub, port 8090), `dozzle-hub` (log viewer, 8088), `backrest` (restic UI, 9898) +- Backup target: `rest-server-ana` on port 8000 + +**NH3 (nh3-docker):** +- `adguard`, `dockge`, plus Beszel/Dozzle agents + +**NH3 (Synology `10.100.50.50`):** +- `rest-server-nh3` — restic backup target (port 8000) + +**ESH home lab (esh-docker-vm):** +- `adguard`, `homeassistant` (macvlan), `esphome`, `mosquitto`, `paperless-ngx`, `pgadmin`, `calibre`, `calibre-web`, `drawio`, `traefik`, `homepage`, `uptime-kuma`, plus Beszel/Dozzle agents + +## Common tasks + +**Refresh one host's snapshot:** +```bash +scripts/refresh-server-info.sh ana-docker +``` + +**Refresh all hosts:** +```bash +scripts/refresh-server-info.sh all +``` + +**Add a new host:** +```bash +scripts/add-host.sh +scripts/refresh-server-info.sh +``` + +**Validate discovery (without hitting the network):** +```bash +scripts/refresh-server-info.sh --validate-only all +``` + +**Push a stack to a host (with diff + confirm):** +```bash +scripts/deploy-stack.sh +``` + +**Pull every server's compose/conf trees into stacks-mirror/ (not committed — see `.gitignore`):** +```bash +scripts/sync-stacks.sh all +``` + +## Backup pipeline + +Backups are driven by per-host `resticprofile` configs under `configs/restic//`, scheduled via systemd timers on each host: + +- **Writes**: each host backs up to its site-local rest-server (`rest-server-ana` or `rest-server-nh3`), over HTTP basic-auth. +- **Authentication**: shared `.htpasswd` file on both rest-servers, one entry per host; credentials stored in `/etc/restic/restic.env` on each client host. +- **Encryption**: per-host client-side passphrase in `/etc/restic/password` (unique per repo; losing it = losing that host's backups). +- **Visibility**: Backrest (`http://10.250.50.70:9898`) shows every repo for browsing/restore. +- **Schedule**: backup at 01:00 daily, `forget` at 03:00 daily, weekly `check --read-data-subset 10%` on Sundays. +- **Prune**: manual ceremony (rest-server runs with `--append-only`, which blocks destructive prune ops). +- **Off-site**: cross-site rsync between the two rest-server data dirs is planned (not yet implemented). + +## Authoritative vs. mirror + +- **Authoritative:** files on each server under `/opt/docker/compose//` and `/opt/docker/conf//`. +- **This workspace:** source-of-truth copies under `stacks//` (hand-curated), and a gitignored mirror under `stacks-mirror/` pulled by `sync-stacks.sh`. + +Edit in `stacks/`, push with `deploy-stack.sh`. Never commit `stacks-mirror/` — it can contain embedded plaintext secrets from upstream compose files that haven't been audited yet. diff --git a/configs/restic/README.md b/configs/restic/README.md new file mode 100644 index 0000000..9b5273a --- /dev/null +++ b/configs/restic/README.md @@ -0,0 +1,118 @@ +# configs/restic + +Per-host restic backup configs, deployed into `/etc/restic/` on each server and driven by `resticprofile` + `systemd` timers. + +## Layout + +``` +configs/restic/ +├── README.md # this file +└── / + ├── profiles.yaml # committed, zero secrets + ├── pre-backup.sh # committed, zero secrets + └── README.md # per-host notes (paths, containers, quirks) +``` + +On each server, deployed to `/etc/restic/`: + +``` +/etc/restic/ +├── profiles.yaml # scp'd from configs/restic//profiles.yaml +├── pre-backup.sh # scp'd, 0755, root:root +├── password # 0400 root:root — client-side encryption passphrase +└── restic.env # 0600 root:root — RESTIC_REPOSITORY=rest:http://user:pw@host:port/path/ +/var/lib/restic/ +├── stage/ # temp staging for DB dumps; owned by root, 0700 +└── last-success # unix timestamp of the last successful run +``` + +## Why this shape + +- **No secrets in committed config.** `profiles.yaml` references `RESTIC_PASSWORD_FILE=/etc/restic/password` and loads `RESTIC_REPOSITORY` from `restic.env`. Both files live only on the host, 0400/0600 root-owned. +- **Creds go in the URL, not netrc.** restic's rest backend doesn't consult `~/.netrc` — HTTP basic-auth has to be embedded in the repository URL. Keeping that URL in an env-file (not the committed YAML) means the secret stays on the host. +- **Pre-hook runs DB dumps into a staging dir**, then `restic backup` includes that dir alongside the regular paths. One snapshot = one point-in-time. +- **`restic forget` is scheduled; `restic prune` is not.** Rest-server's `--append-only` blocks prune from the client side by design. Prune is a manual ceremony (flip the flag, run prune, flip back). + +## Install restic + resticprofile on each host + +```bash +# Current-enough restic. Debian 12 ships 0.14 (too old for some flags); +# Debian 13 ships 0.18. If you're on 12, grab the .deb from the upstream +# release page instead. +sudo apt install -y restic # or install 0.18+ from github.com/restic/restic/releases + +# resticprofile is not in Debian. Download the .deb from its release page. +v=$(curl -sI https://github.com/creativeprojects/resticprofile/releases/latest \ + | awk -F'/' '/^location:/{sub(/\r/,""); print $NF}') +v=${v#v} +url="https://github.com/creativeprojects/resticprofile/releases/download/v${v}/resticprofile_${v}_linux_amd64.deb" +curl -sL -o /tmp/resticprofile.deb "$url" +sudo dpkg -i /tmp/resticprofile.deb +rm /tmp/resticprofile.deb +resticprofile version +``` + +## Per-host deploy flow (done once per host) + +```bash +HOST=ana-docker # or ana-ml2, nh3-docker, esh-docker-vm + +# 1. Create the target dir (one-time) +ssh -t "$HOST" 'sudo install -d -o root -g root -m 0755 /etc/restic' +ssh -t "$HOST" 'sudo install -d -o root -g root -m 0700 /var/lib/restic/stage' + +# 2. Seed the two secret files on the host (never in this repo): +# - the client-side encryption passphrase (the one used at `restic init`) +# - an env-file with the full RESTIC_REPOSITORY URL including HTTP creds +ssh -t "$HOST" 'sudo install -o root -g root -m 0400 /dev/null /etc/restic/password' +ssh -t "$HOST" 'sudo install -o root -g root -m 0600 /dev/null /etc/restic/restic.env' + +# Seed content (replace <…> with real values from your vault): +ssh -t "$HOST" "echo '' | sudo tee /etc/restic/password >/dev/null" +ssh -t "$HOST" "echo 'RESTIC_REPOSITORY=rest:http://:@:8000//' | sudo tee /etc/restic/restic.env >/dev/null" + +# 3. Push the committed config + pre-hook +scp "configs/restic/$HOST/profiles.yaml" "$HOST:/tmp/profiles.yaml" +scp "configs/restic/$HOST/pre-backup.sh" "$HOST:/tmp/pre-backup.sh" +ssh -t "$HOST" " + sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml + sudo install -o root -g root -m 0755 /tmp/pre-backup.sh /etc/restic/pre-backup.sh + rm -f /tmp/profiles.yaml /tmp/pre-backup.sh +" + +# 4. Test the profile before scheduling +ssh -t "$HOST" 'sudo resticprofile --config /etc/restic/profiles.yaml --name default backup --dry-run' + +# 5. When dry-run looks clean, wire up systemd timers +ssh -t "$HOST" 'sudo resticprofile --config /etc/restic/profiles.yaml schedule' +# → installs restic-backup@.{service,timer} units for backup/forget/check + +# 6. Verify +ssh -t "$HOST" 'sudo systemctl list-timers | grep restic' +``` + +## Prune ceremony (quarterly or as needed) + +Per-repo, when enough forgotten-but-still-on-disk snapshots accumulate: + +```bash +# On the rest-server host (ana-docker or the Synology): +# 1. Stop the append-only rest-server, start one without the flag +# (simplest: edit the stack's .env, remove --append-only from OPTIONS, +# docker compose up -d) + +# 2. From the client host, run prune +ssh -t 'sudo resticprofile --config /etc/restic/profiles.yaml --name default prune' + +# 3. Put --append-only back on the rest-server and docker compose up -d. +``` + +Alternatively stand up a second rest-server stack on port 8001 without `--append-only` and point prune runs at that endpoint; keep the append-only one for daily writes. + +## Monitoring + +Each successful run writes `/var/lib/restic/last-success` with the current unix timestamp (this is the `post-backup` hook in `profiles.yaml`). Beszel can be configured to alert when that file's mtime exceeds ~36 hours. + +```bash +ssh 'stat -c "%y %n" /var/lib/restic/last-success 2>/dev/null || echo no successful run yet' +``` diff --git a/configs/restic/ana-docker/pre-backup.sh b/configs/restic/ana-docker/pre-backup.sh new file mode 100755 index 0000000..5c5d9ba --- /dev/null +++ b/configs/restic/ana-docker/pre-backup.sh @@ -0,0 +1,75 @@ +#!/bin/bash +# pre-backup.sh — ana-docker. +# Runs as root from resticprofile's `run-before`, before `restic backup`. +# +# Produces DB dumps in /var/lib/restic/stage/ so the nightly restic +# snapshot captures consistent point-in-time data for services whose +# raw volume files are not safe to back up live. +# +# Containers handled here: +# - synapse-db (Postgres 16) +# - seafile-mysql (MariaDB 10.6) +# - vaultwarden (SQLite w/ WAL; online .backup via sqlite3 if available) +# +# Gitea's DB is external (hosted elsewhere in the LAN) so we only back +# up its data volume; whoever owns the gitea Postgres backs it up +# separately. +# +# Idempotent: clears and recreates its staging files each run. + +set -euo pipefail + +STAGE=/var/lib/restic/stage +install -d -o root -g root -m 0700 "$STAGE" + +log() { printf '%s pre-backup(ana-docker): %s\n' "$(date -Is)" "$*"; } + +# Purge previous stage so stale dumps don't pile up and end up in the snapshot. +find "$STAGE" -mindepth 1 -maxdepth 1 -exec rm -rf {} + + +# ---------- synapse-db (Postgres) --------------------------------------------- +if docker inspect synapse-db >/dev/null 2>&1; then + log "dumping synapse postgres" + # -Fc custom format, internally compressed + restore-to-subset friendly + docker exec synapse-db \ + pg_dump -U synapse -d synapse -Fc --clean --if-exists \ + > "$STAGE/synapse.pg_dump" +else + log "skip synapse: container not present" +fi + +# ---------- seafile-mysql (MariaDB) ------------------------------------------- +if docker inspect seafile-mysql >/dev/null 2>&1; then + log "dumping seafile mariadb" + # The root password lives in the container's own env (MYSQL_ROOT_PASSWORD); + # expand it inside the container so it never lands in the host's process list. + docker exec seafile-mysql sh -c \ + 'mysqldump -uroot -p"$MYSQL_ROOT_PASSWORD" --all-databases --single-transaction --quick 2>/dev/null' \ + | gzip -c > "$STAGE/seafile.sql.gz" +else + log "skip seafile: container not present" +fi + +# ---------- vaultwarden (SQLite + WAL) ---------------------------------------- +# Vaultwarden uses SQLite in WAL mode. A live copy of db.sqlite3 + -wal is +# usually recoverable, but sqlite3's own .backup pragma is the correct way +# to get a consistent snapshot. If the vaultwarden image has sqlite3 +# available, use it; otherwise rely on restic backing up the raw volume. +if docker inspect vaultwarden >/dev/null 2>&1; then + if docker exec vaultwarden sh -c 'command -v sqlite3 >/dev/null 2>&1'; then + log "dumping vaultwarden sqlite via .backup" + docker exec vaultwarden sqlite3 /data/db.sqlite3 \ + ".backup /tmp/vaultwarden.sqlite3" + docker cp vaultwarden:/tmp/vaultwarden.sqlite3 "$STAGE/vaultwarden.sqlite3" + docker exec vaultwarden rm -f /tmp/vaultwarden.sqlite3 + else + log "skip vaultwarden .backup: sqlite3 not in container (raw volume still included via restic)" + fi +else + log "skip vaultwarden: container not present" +fi + +# ---------- summary ----------------------------------------------------------- +size=$(du -sh "$STAGE" 2>/dev/null | awk '{print $1}') +count=$(find "$STAGE" -type f | wc -l) +log "stage ready: $count files, $size total" diff --git a/configs/restic/ana-docker/profiles.yaml b/configs/restic/ana-docker/profiles.yaml new file mode 100644 index 0000000..726c4c6 --- /dev/null +++ b/configs/restic/ana-docker/profiles.yaml @@ -0,0 +1,76 @@ +# resticprofile config for ana-docker. +# +# Writes to the Anaheim-side rest-server at 10.250.50.70 as user +# `ana-docker`. The full REST URL (including HTTP basic-auth creds) +# lives in /etc/restic/restic.env — loaded via env-file so this YAML +# carries zero secrets and is safe to version-control. +# +# The client-side encryption passphrase lives in /etc/restic/password. + +version: "1" + +global: + priority: low + ionice: true + ionice-class: 2 + ionice-level: 7 + min-memory: 100 + +default: + env-file: /etc/restic/restic.env # provides RESTIC_REPOSITORY=rest:http://user:pw@… + env: + RESTIC_PASSWORD_FILE: /etc/restic/password + initialize: false # repo was created by `restic init` + lock: /var/lock/restic-ana-docker.lock + + backup: + verbose: 1 + run-before: + - /etc/restic/pre-backup.sh + run-after: + - date +%s > /var/lib/restic/last-success + source: + - /opt/docker + - /var/lib/docker/volumes + - /var/lib/restic/stage + exclude: + # Docker internals we never want in a backup + - /var/lib/docker/volumes/backingFsBlockDev + - /var/lib/docker/volumes/metadata.db + # Raw DB files — we dump them via pre-backup.sh into /var/lib/restic/stage + - /var/lib/docker/volumes/synapse-db-data + - /var/lib/docker/volumes/synapse_synapse-db-data + - /var/lib/docker/volumes/seafile_db + # Ephemeral / regenerable junk + - /opt/docker/compose/*/logs + - /opt/docker/conf/traefik-ana/acme.json # secret material; excluded everywhere + - /opt/docker/conf/crowdsec/hub # upstream-managed, regenerable + - "**/*.log" + - "**/*.log.*" + - "**/*.pid" + tag: + - host:ana-docker + - site:ana + - fleet:pfi + schedule: "*-*-* 01:00:00" + schedule-permission: system + schedule-log: /var/log/restic-backup.log + + forget: + keep-daily: 7 + keep-weekly: 4 + keep-monthly: 12 + keep-yearly: 3 + # NOTE: no `prune: true` — rest-server runs with --append-only, which + # blocks the destructive half of prune. See README.md "Prune ceremony". + tag: + - host:ana-docker + schedule: "*-*-* 03:00:00" + schedule-permission: system + schedule-log: /var/log/restic-forget.log + + check: + read-data-subset: 10% + schedule: "Sun *-*-* 05:00:00" + schedule-permission: system + schedule-log: /var/log/restic-check.log diff --git a/docs/pfi/chromadb-setup.md b/docs/pfi/chromadb-setup.md new file mode 100644 index 0000000..4ed12e1 --- /dev/null +++ b/docs/pfi/chromadb-setup.md @@ -0,0 +1,146 @@ +# ChromaDB Setup Documentation + +**Project**: Infrastructure-PFI +**Target Server**: PFI-ANA-Docker (VM 102) +**IP Address**: 10.250.50.x (VLAN 50) +**Status**: Ready for deployment + +## Overview + +ChromaDB is an embedded vector database optimized for AI/ML applications. This deployment provides: + +- Persistent vector storage on `/tank/chromadb/` +- REST API on port 8000 (internal + Traefik-routed) +- Token-based authentication +- Automated backup and health monitoring + +## Architecture + +``` +┌──────────────────────────────────────────────────────────┐ +│ PFI-ANA-Docker (VM 102) │ +│ │ +│ ┌──────────────┐ ┌──────────────┐ ┌───────────┐ │ +│ │ Traefik │───▶│ ChromaDB │ │ Dockge │ │ +│ │ Reverse │ │ (Port 8000) │ │ Manager │ │ +│ │ Proxy │ │ │ │ │ │ +│ └──────────────┘ └──────┬───────┘ └───────────┘ │ +│ │ │ +│ ┌────────▼────────┐ │ +│ │ /tank/chromadb │ │ +│ │ (bind mount) │ │ +│ └─────────────────┘ │ +└──────────────────────────────────────────────────────────┘ +``` + +## File Locations (on VM 102) + +| Host Path | Purpose | +|---|---| +| `/opt/docker/conf/chromadb/` | Compose file, auth token, config | +| `/opt/docker/conf/chromadb/docker-compose.yml` | Main compose file | +| `/opt/docker/conf/chromadb/auth_token` | Token for API authentication | +| `/tank/chromadb/` | Persistent vector data (bind mount) | +| `/opt/docker/backups/chromadb/` | Backup archives | + +## Authentication + +This deployment uses **ChromaDB's native token auth**: + +- A random 64-hex-char token is generated during setup (`openssl rand -hex 32`) +- The token is stored at `/opt/docker/conf/chromadb/auth_token` (mode 600) +- Clients must supply the token via `Settings`: + +```python +import chromadb +from chromadb.config import Settings + +client = chromadb.HttpClient( + host="10.250.50.x", # or chromadb.pfi.local via Traefik + port=8000, + settings=Settings( + chroma_client_auth_provider="chromadb.auth.token.TokenAuthClientProvider", + chroma_client_auth_credentials="YOUR_TOKEN_HERE", + ), +) +print(client.heartbeat()) +``` + +## Deployment Steps + +### 1. Copy compose file to VM 102 + +```bash +scp configs/pfi-ana/docker/compose-examples/chromadb/docker-compose.yml \ + root@10.250.50.x:/opt/docker/conf/chromadb/docker-compose.yml +``` + +### 2. Run the setup script (on VM 102) + +```bash +# Copy scripts to VM 102 +scp scripts/setup-chromadb.sh root@10.250.50.x:/opt/docker/conf/chromadb/ +ssh root@10.250.50.x + +# Run setup +cd /opt/docker/conf/chromadb +chmod +x setup-chromadb.sh +./setup-chromadb.sh +``` + +### 3. Verify + +```bash +curl http://localhost:8000/api/v1/health +``` + +### 4. (Optional) Run the demo + +```bash +# Copy demo files +scp -r configs/pfi-ana/docker/compose-examples/chromadb/ root@10.250.50.x:/tmp/chromadb-demo/ + +# On VM 102, edit CHROMA_TOKEN in docker-compose.demo.yml +cd /tmp/chromadb-demo +# Set CHROMA_TOKEN in docker-compose.demo.yml to match auth_token +docker compose -f docker-compose.demo.yml up +``` + +## Monitoring & Maintenance + +| Task | Command | +|---|---| +| Health check | `./scripts/health-check-chromadb.sh` | +| View logs | `docker logs -f chromadb` | +| Backup | `./scripts/backup-chromadb.sh` | +| Restart | `docker compose restart` | +| Stop | `docker compose down` | + +### Cron (daily backup at 2 AM) + +```cron +0 2 * * * /opt/docker/conf/chromadb/backup-chromadb.sh >> /var/log/chromadb-backup.log 2>&1 +``` + +## Networking + +| Aspect | Value | +|---|---| +| Docker network | `traefik-net` (aliased as `tnet`) | +| Internal port | 8000 | +| Traefik host rule | `chromadb.pfi.local` | +| Traefik entrypoint | `websecure` (HTTPS) | +| TLS | Enabled via Traefik | + +## Project Files + +| File | Purpose | +|---|---| +| `configs/pfi-ana/docker/compose-examples/chromadb/docker-compose.yml` | Production compose (for Dockge) | +| `configs/pfi-ana/docker/compose-examples/chromadb/docker-compose.demo.yml` | Demo client | +| `configs/pfi-ana/docker/compose-examples/chromadb/Dockerfile.demo` | Demo image | +| `configs/pfi-ana/docker/compose-examples/chromadb/scripts/demo.py` | Demo test script | +| `scripts/setup-chromadb.sh` | Deployment script (run on VM 102) | +| `scripts/backup-chromadb.sh` | Backup script (run on VM 102) | +| `scripts/health-check-chromadb.sh` | Health monitoring (run on VM 102) | +| `scripts/quickstart-chromadb.sh` | Convenience wrapper for setup | diff --git a/docs/pfi/docker-stack.md b/docs/pfi/docker-stack.md new file mode 100644 index 0000000..35765d9 --- /dev/null +++ b/docs/pfi/docker-stack.md @@ -0,0 +1,333 @@ +# PFI-ANA Docker Stack + +## Overview + +PFI-ANA (the Colo) runs Docker services managed through **Dockge**, a compose-aware Docker management UI. All services that require inbound HTTP/HTTPS routing join a shared external Docker network called `traefik-net`, allowing **Traefik** to act as a reverse proxy and handle TLS termination and routing. + +## Conventions + +### Network + +| Network | Docker Name | Purpose | +|---|---|---| +| Traefik network | `traefik-net` | Shared external network. Services join as `tnet` so Traefik can discover them. | + +Every compose file that needs to be reachable through Traefik **must** include: + +```yaml +networks: + tnet: + name: traefik-net + external: true +``` + +And the service must list `tnet` under its `networks` key. + +### Storage Paths (Host) + +| Host Path | Purpose | +|---|---| +| `/opt/docker/compose//` | Per-service compose files (managed by Dockge) | +| `/opt/docker/conf//` | Per-service configuration files (bind-mounted into containers) | +| `/tank/` | Large / persistent data storage (e.g., AI models, generated images, voice data) | + +### GPU Support + +Services requiring GPU access use the **NVIDIA Container Toolkit**: + +```yaml +runtime: nvidia +``` + +or the more explicit device reservation: + +```yaml +deploy: + resources: + reservations: + devices: + - driver: nvidia + count: all + capabilities: [gpu] +``` + +### Homepage Dashboard Labels + +Several services include Docker labels for **Homepage** (a dashboard UI). The convention is: + +```yaml +labels: + - homepage.group= + - homepage.name= + - homepage.icon= + - homepage.description= + - homepage.href=http://: +``` + +All services reference the VM 102 host IP `10.250.50.70` (PFI-ANA_DOCKER). + +### Compose File Location + +Dockge expects compose files under `/opt/docker/compose/` on the Docker host. The live compose files are version-controlled in this project under `configs/pfi-ana/docker/compose/`. Configuration files that containers bind-mount live under `configs/pfi-ana/docker/conf/`. + +--- + +## Live Services Index + +| Service | Host Port | Container Port | GPU | Homepage Group | Status | Compose File | Config File | +|---|---|---|---|---|---|---|---| +| **Dockge** | 5001 | 5001 | No | PFI-ANA | ✅ Live | `compose/dockge/compose.yaml` | — | +| **llama-swap** | 9292 | 8080 | Yes (CUDA) | — | ✅ Live | `compose/llama-swap/compose.yaml` | `conf/llama-swap/config.yaml` | +| **ComfyUI** | 8188 | 8188 | Yes (all caps) | AI Systems | ✅ Live | `compose/comfyui/compose.yaml` | — | +| **VibeVoice** | 8745 | 8745 | Yes (gpu) | AI Systems | ✅ Live | `compose/vibevoice/compose.yaml` | — | +| **Parakeet STT** | 8300 | 8000 | Yes (gpu) | AI Systems | ✅ Live | `compose/parakeet/compose.yaml` | — | +| **ChromaDB** | 8000 | 8000 | No | AI Systems | ✅ Live | `compose/chromadb/compose.yaml` | `/opt/docker/conf/chromadb/auth_token` | + +> All paths relative to `configs/pfi-ana/docker/`. + +--- + +## Service Details + +### 1. Dockge — Docker Compose Management UI + +- **Image**: `louislam/dockge:latest` +- **Port**: 5001 → 5001 +- **Restart policy**: `unless-stopped` +- **Homepage group**: PFI-ANA +- **Compose file**: `compose/dockge/compose.yaml` + +**Volumes**: +| Host / Volume | Container | Purpose | +|---|---|---| +| `/var/run/docker.sock` | `/var/run/docker.sock` | Docker socket for managing containers | +| `dockge_data` (named volume) | `/app/data` | Dockge application data | +| `/opt/docker/compose` | `/opt/docker/compose` | Compose stack directory | + +**Environment**: +- `DOCKGE_STACKS_DIR=/opt/docker/compose` — tells Dockge where to find/manage compose stacks + +**Notes**: Dockge is the management interface for all other compose stacks on this host. It has full Docker daemon access via the socket mount. + +--- + +### 2. llama-swap — Multi-Model LLM Gateway + +- **Image**: `ghcr.io/mostlygeek/llama-swap:cuda` +- **Port**: 9292 → 8080 +- **Runtime**: `nvidia` (CUDA) +- **Compose file**: `compose/llama-swap/compose.yaml` +- **Config file**: `conf/llama-swap/config.yaml` +- **Interactive**: `stdin_open: true`, `tty: true` (required by llama-swap) + +**Volumes**: +| Host Path | Container Path | Purpose | +|---|---|---| +| `/opt/docker/conf/llama-swap/config.yaml` | `/app/config.yaml` | llama-swap configuration (models, groups, params) | +| `/tank/aimodels/llm` | `/models` | LLM model files (GGUF format) | + +**Configured Models** (from `config.yaml`): + +| Model ID | Display Name | Quantization | Context Size | TTL (s) | Notes | +|---|---|---|---|---|---| +| `qwen3-4b` | Qwen3-4B-Instruct-2507-Q6_K | Q6_K | default | 0 (persistent) | Small general-purpose model | +| `glm4.5-air` | GLM-4.5-Air Q4_K_M | Q4_K_M | 40,000 | 600 | Flash attention enabled | +| `skyfall-r1-31b-q6k` | Skyfall 31B v4 | Q6_K_L | 40,000 | 600 | Flash attention, full GPU offload | +| `GLM-Steam-106B-QK4M-A12B` | GLM-Steam 106B A12B | Q4_K_M | 40,000 | 600 | 2-shard model, MoE with 12B active | +| `kimik2-q2kxl` | Kimi K2 Instruct | UD-Q2_K_XL | default | 600 | 8-shard model, only 2 GPU layers (CPU-heavy) | +| `qwen3-coder-30b-iq4-nl` | Qwen3 Coder 30B A3B | IQ4_NL | 40,000 | 0 (persistent) | MoE 3B active, coding-optimized | +| `unsloth-granite-4-small` | Granite 4.0 Small | Q4_K_M | 120,000 | 0 (persistent) | IBM Granite, deterministic (temp=0) | +| `qwen3.5-35-a3b` | Qwen 3.5 35B A3B | UD-Q4_K_XL | 32,768 | 0 (persistent) | MoE, thinking mode, temp=1.0 | +| `qwen3.5-35-a3b-code` | Qwen 3.5 35B A3B Code | UD-Q4_K_XL | 32,768 | 0 (persistent) | Same model, code-tuned params (temp=0.6) | +| `gemma4-26b-a4b` | Gemma 4 26B A4B | UD-Q4_K_XL | 32,768 | 600 | MoE 4B active, thinking enabled, supports images | +| `gemma4-31b-dense` | Gemma 4 31B Dense | UD-Q4_K_XL | 32,768 | 600 | Full dense model, thinking enabled, supports images | +| `embeddinggemma-300M` | Embedding Gemma 300M | Q8_0 | 2,048 | 0 (persistent) | Embedding model, cls pooling | +| `qwen3-embedding-0.6B` | Qwen3 Embedding 0.6B | Q8_0 | 32,768 | 0 (persistent) | Embedding model, mean pooling | +| `jina-reranker-v3-0.6B` | Jina Reranker v3 | Q8_0 | 32,768 | 0 (persistent) | Reranking model | +| `bge-reranker-v2-m3-0.6B` | BGE Reranker v2 m3 | Q8_0 | 32,768 | 0 (persistent) | Reranking model | + +**Model Groups**: + +| Group | Swap | Exclusive | Persistent | Members | +|---|---|---|---|---| +| `high-reasoning` | false | false | — | qwen3.5-35-a3b, qwen3.5-35-a3b-code, gemma4-31b-dense | +| `utility` | false | false | ✅ | embeddinggemma-300M, bge-reranker-v2-m3-0.6B | + +**Global Settings**: +- `healthCheckTimeout`: 1200 seconds (20 minutes) — long timeout for large models +- `logLevel`: info +- `metricsMaxInMemory`: 1000 + +--- + +### 3. ComfyUI — Image Generation UI + +- **Image**: `mmartial/comfyui-nvidia-docker:ubuntu24_cuda13.0-latest` +- **Port**: 8188 → 8188 +- **Runtime**: `nvidia` with full device reservation (gpu, compute, utility capabilities) +- **Restart policy**: `unless-stopped` +- **Homepage group**: AI Systems +- **Compose file**: `compose/comfyui/compose.yaml` + +**Volumes**: +| Host Path | Container Path | Purpose | +|---|---|---| +| `/tank/comfy/run` | `/comfy/mnt` | ComfyUI workspace / output directory | +| `/tank/aimodels/img/comfy` | `/basedir` | Image models and ComfyUI base directory | + +**Environment**: +| Variable | Value | Purpose | +|---|---|---| +| `WANTED_UID` | 1001 | Run as user ID 1001 | +| `WANTED_GID` | 1002 | Run as group ID 1002 | +| `BASE_DIRECTORY` | /basedir | ComfyUI base directory path | +| `SECURITY_LEVEL` | weak | Relaxed security (private network) | +| `NVIDIA_VISIBLE_DEVICES` | all | Expose all GPUs | +| `NVIDIA_DRIVER_CAPABILITIES` | all | Enable all GPU capabilities | + +**Notes**: Runs with user-mapped permissions (UID 1001 / GID 1002). The `basedir` points to the image model storage on `/tank`. + +--- + +### 4. VibeVoice — Voice/Audio AI Service + +- **Image**: `eworkerinc/vibevoice:latest` +- **Container name**: `vibevoice` +- **Port**: 8745 → 8745 +- **GPU**: Yes (all devices, gpu capability) +- **Restart policy**: `unless-stopped` +- **Homepage group**: AI Systems +- **Compose file**: `compose/vibevoice/compose.yaml` + +**Volumes**: +| Host Path | Container Path | Purpose | +|---|---|---| +| `/tank/vibevoice/hf` | `/root/.cache/huggingface` | HuggingFace model cache | +| `/tank/vibevoice/voices` | `/app/voices` | Voice data / presets | +| `/tank/vibevoice/state` | `/var/lib/eworker` | Application state persistence | + +**Environment**: +| Variable | Value | Purpose | +|---|---|---| +| `ENABLE_1_5B` | true | Enable 1.5B parameter voice model | +| `ENABLE_LARGE` | true | Enable large voice model | +| `AUTH_REQUIRED` | true | Require authentication | +| `CORS_ENABLED` | true | Enable CORS headers | +| `ALLOWED_ORIGINS` | * | Allow all origins (development/private network) | + +--- + +### 5. Parakeet STT — Speech-to-Text Service + +- **Image**: `parakeet-stt` (locally built) +- **Port**: 8300 → 8000 +- **GPU**: Yes (all devices, gpu capability) +- **Restart policy**: `unless-stopped` +- **Homepage group**: AI Systems +- **Compose file**: `compose/parakeet/compose.yaml` +- **Env file**: `.env` (not tracked in project — likely contains API keys or model config) + +**Volumes**: +| Volume | Container Path | Purpose | +|---|---|---| +| `parakeet_cache` (named volume) | `/root/.cache` | Model download cache | + +**Notes**: Uses a locally-built image (no registry prefix). The `.env` file is referenced but not stored in the project — it likely contains environment-specific configuration on the Docker host. + +--- + +### 6. ChromaDB — Vector Database + +- **Image**: `chromadb/chroma:latest` +- **Container name**: `chromadb` +- **Port**: 8000 → 8000 +- **GPU**: No +- **Restart policy**: `unless-stopped` +- **Homepage group**: AI Systems +- **Compose file**: `compose/chromadb/compose.yaml` + +**Volumes**: +| Host Path | Container Path | Mode | Purpose | +|---|---|---|---| +| `/opt/docker/conf/chromadb` | `/conf` | read-only | Config directory (contains auth_token) | +| `/tank/chromadb` | `/data` | read-write | Persistent vector data | + +**Environment**: +| Variable | Value | Purpose | +|---|---|---| +| `CHROMA_SERVER_AUTHN_CREDENTIALS_FILE` | `/conf/auth_token` | Path to auth token file inside container | +| `CHROMA_SERVER_AUTHN_PROVIDER` | `chromadb.server.auth.token.TokenAuthenticationServerProvider` | Enable token-based authentication | +| `IS_PERSISTENT` | TRUE | Enable persistent storage | +| `PERSIST_DIRECTORY` | `/data` | Where vector data is stored inside container | +| `ANONYMIZED_TELEMETRY` | FALSE | Disable telemetry | + +**Health Check**: +| Setting | Value | +|---|---| +| Test | `curl -f http://localhost:8000/api/v1/health` | +| Interval | 30s | +| Timeout | 10s | +| Retries | 3 | +| Start period | 40s | + +**Authentication**: Token-based. The auth token is stored at `/opt/docker/conf/chromadb/auth_token` on the host (mode 600), generated with `openssl rand -hex 32`. Clients must supply this token to access the API. + +**Notes**: CPU-only service (no GPU). The compose-examples directory contains a reference compose file with Traefik labels and a demo app — see [chromadb-setup.md](chromadb-setup.md) for full deployment instructions and the demo. + +--- + +## Infrastructure Summary + +``` +┌──────────────────────────────────────────────────────────────┐ +│ PFI-ANA (10.250.50.70) │ +│ Docker Host (PFI-ANA_DOCKER) │ +│ │ +│ ┌──────────┐ Manages all compose stacks │ +│ │ Dockge │◄─── /opt/docker/compose/* │ +│ │ :5001 │ /var/run/docker.sock │ +│ └──────────┘ │ +│ │ +│ ┌──────────────────┐ GPU via passthrough │ +│ │ llama-swap :9292 (CUDA, multi-model gateway) │ │ +│ │ ├── 15 models (chat, code, embedding, reranker) │ │ +│ │ ├── 2 groups (high-reasoning, utility) │ │ +│ │ └── Models from /tank/aimodels/llm │ │ +│ └──────────────────┘ GPU via passthrough │ +│ │ +│ ┌──────────┐ ┌──────────┐ ┌──────────┐ GPU via │ +│ │ ComfyUI │ │ VibeVoice │ │ Parakeet │ passthrough │ +│ │ :8188 │ │ :8745 │ │ :8300 │ │ +│ │ (GPU, img) │ │ (GPU, voice)│ │ (GPU, STT) │ │ +│ └──────────┘ └──────────┘ └──────────┘ │ +│ │ +│ ┌──────────┐ │ +│ │ ChromaDB │ Token auth, persistent vectors │ +│ │ :8000 │ /tank/chromadb (data) │ +│ │ (CPU only) │ /opt/docker/conf/chromadb (config) │ +│ └──────────┘ │ +│ │ +│ ── All services on traefik-net (external) ── │ +│ ── /tank/* = persistent large data storage ── │ +│ ── /opt/docker/* = config + compose files ── │ +└──────────────────────────────────────────────────────────────┘ +``` + +## Port Allocation + +| Port | Service | Protocol | +|---|---|---| +| 5001 | Dockge | HTTP | +| 8000 | ChromaDB | HTTP | +| 8188 | ComfyUI | HTTP | +| 8300 | Parakeet STT | HTTP (→ container 8000) | +| 8745 | VibeVoice | HTTP | +| 9292 | llama-swap | HTTP (→ container 8080) | + +## Example Configurations + +Reference/example compose files (not live) are stored in `configs/pfi-ana/docker/compose-examples/`: +- `compose-examples/llama-swap/docker-compose.yml` — earlier llama-swap reference +- `compose-examples/chromadb/` — ChromaDB with Traefik labels, auth, health checks, and demo app + +Full ChromaDB setup instructions: [chromadb-setup.md](chromadb-setup.md) diff --git a/docs/pfi/model-list.md b/docs/pfi/model-list.md new file mode 100644 index 0000000..5bd5a56 --- /dev/null +++ b/docs/pfi/model-list.md @@ -0,0 +1,81 @@ +# PFI-ANA Model Inventory +# Generated from: find /models -maxdepth 3 -name "*.gguf" | sort +# Synchronized with llama-swap config.yaml on 2025-07-18 +# +# Status Key: +# [ACTIVE] = configured in llama-swap +# [PENDING] = on disk, not yet in config (or still downloading) +# [REMOVED] = was in old config, not on disk +# +# ============================================================================ +# KB Optimization Pass (2025-07-18): +# - Qwen 3.5: added presence_penalty, thinking mode enable_thinking +# - Qwen3-Coder-Next: temp 0.3→1.0, top-p 0.85→0.95, top-k 20→40, min-p 0→0.01 +# - Gemma 4: added repeat_penalty 1.0, thinking mode enable_thinking +# - Nemotron: already applied in prior pass (--special, min-p 0.01, seed 3407) +# ============================================================================ + +[ACTIVE] ./ggml-org_Qwen3-Reranker-0.6B-Q8_0-GGUF/qwen3-reranker-0.6b-q8_0.gguf + +[ACTIVE] ./unsloth_Qwen3.5-122B-A10B-GGUF/UD-Q4_K_XL/Qwen3.5-122B-A10B-UD-Q4_K_XL-00001-of-00003.gguf +[ACTIVE] ./unsloth_Qwen3.5-122B-A10B-GGUF/UD-Q4_K_XL/Qwen3.5-122B-A10B-UD-Q4_K_XL-00002-of-00003.gguf +[ACTIVE] ./unsloth_Qwen3.5-122B-A10B-GGUF/UD-Q4_K_XL/Qwen3.5-122B-A10B-UD-Q4_K_XL-00003-of-00003.gguf + +[ACTIVE] ./ibm-granite_granite-4.0-micro-GGUF/granite-4.0-micro-Q4_K_M.gguf + +[ACTIVE] ./unsloth_GLM-4.7-Flash-GGUF/GLM-4.7-Flash-UD-Q4_K_XL.gguf + +[ACTIVE] ./jinaai_jina-reranker-v3-GGUF/jina-reranker-v3-Q8_0.gguf + +[ACTIVE] ./unsloth_NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_XL/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-00001-of-00003.gguf +[ACTIVE] ./unsloth_NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_XL/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-00002-of-00003.gguf +[ACTIVE] ./unsloth_NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_XL/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-00003-of-00003.gguf + +[ACTIVE] ./bartowski_TheDrummer_Skyfall-31B-v4-GGUF/TheDrummer_Skyfall-31B-v4-Q6_K_L.gguf + +[ACTIVE] ./RP/BeaverAI_Skyfall-R1-31B-v4a-GGUF/Skyfall-R1-31B-v4a-Q6_K.gguf + +[ACTIVE] ./RP/bartowski_TheDrummer_GLM-Steam-106B-A12B-v1-GGUF/TheDrummer_GLM-Steam-106B-A12B-v1-Q4_K_M-00001-of-00002.gguf +[ACTIVE] ./RP/bartowski_TheDrummer_GLM-Steam-106B-A12B-v1-GGUF/TheDrummer_GLM-Steam-106B-A12B-v1-Q4_K_M-00002-of-00002.gguf + +[ACTIVE] ./unsloth_Qwen3.5-9B-GGUF/Qwen3.5-9B-UD-Q4_K_XL.gguf + +[ACTIVE] ./unsloth_granite-4.0-h-small-GGUF/granite-4.0-h-small-Q4_K_M.gguf + +[ACTIVE] ./unsloth_Kimi-K2-Instruct-0905-GGUF/UD-Q2_K_XL/Kimi-K2-Instruct-0905-UD-Q2_K_XL-00001-of-00008.gguf +[ACTIVE] ./unsloth_Kimi-K2-Instruct-0905-GGUF/UD-Q2_K_XL/Kimi-K2-Instruct-0905-UD-Q2_K_XL-00002-of-00008.gguf +[ACTIVE] ./unsloth_Kimi-K2-Instruct-0905-GGUF/UD-Q2_K_XL/Kimi-K2-Instruct-0905-UD-Q2_K_XL-00003-of-00008.gguf +[ACTIVE] ./unsloth_Kimi-K2-Instruct-0905-GGUF/UD-Q2_K_XL/Kimi-K2-Instruct-0905-UD-Q2_K_XL-00004-of-00008.gguf +[ACTIVE] ./unsloth_Kimi-K2-Instruct-0905-GGUF/UD-Q2_K_XL/Kimi-K2-Instruct-0905-UD-Q2_K_XL-00005-of-00008.gguf +[ACTIVE] ./unsloth_Kimi-K2-Instruct-0905-GGUF/UD-Q2_K_XL/Kimi-K2-Instruct-0905-UD-Q2_K_XL-00006-of-00008.gguf +[ACTIVE] ./unsloth_Kimi-K2-Instruct-0905-GGUF/UD-Q2_K_XL/Kimi-K2-Instruct-0905-UD-Q2_K_XL-00007-of-00008.gguf +[ACTIVE] ./unsloth_Kimi-K2-Instruct-0905-GGUF/UD-Q2_K_XL/Kimi-K2-Instruct-0905-UD-Q2_K_XL-00008-of-00008.gguf + +[PENDING] ./unsloth_Qwen3-Coder-Next-GGUF/Qwen3-Coder-Next-UD-Q4_K_XL.gguf + NOTE: Download may be incomplete (has .incomplete files). Config entry ready. + +[ACTIVE] ./unsloth_Nemotron-3-Nano-30B-A3B-GGUF/Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf + +[ACTIVE] ./unsloth_gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf + +[ACTIVE] ./unsloth_Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf + +[ACTIVE] ./Qwen_Qwen3-Embedding-0.6B-GGUF/Qwen3-Embedding-0.6B-Q8_0.gguf + +[ACTIVE] ./unsloth_gemma-4-31B-it-GGUF/gemma-4-31B-it-UD-Q4_K_XL.gguf + +[ACTIVE] ./ggml-org_embeddinggemma-300M-GGUF/embeddinggemma-300M-Q8_0.gguf + +# ============================================================================ +# MODELS REMOVED FROM CONFIG (not on disk or superseded): +# ============================================================================ +# [REMOVED] qwen3-4b — Qwen3-4B-Instruct-2507-Q6_K (not on disk) +# [REMOVED] glm4.5-air — GLM-4.5-Air-Q4_K_M (not on disk, superseded by glm4.7-flash) +# [REMOVED] qwen3-coder-30b — Qwen3-Coder-30B-A3B (not on disk, superseded by qwen3-coder-next) +# [REMOVED] bge-reranker-v2-m3 — Replaced by qwen3-reranker-0.6B +# [FIXED] glm-steam-106b — Path fixed: was missing RP/ prefix +# ============================================================================ +# STILL DOWNLOADING (not yet configured): +# ============================================================================ +# [PENDING] Hermes-4-14B — Multiple quant downloads in progress (.incomplete files) +# ============================================================================ diff --git a/docs/pfi/proxmox-vms.md b/docs/pfi/proxmox-vms.md new file mode 100644 index 0000000..24b4475 --- /dev/null +++ b/docs/pfi/proxmox-vms.md @@ -0,0 +1,260 @@ +# PFI-ANA Proxmox VM Inventory + +**Hypervisor**: Proxmox VE at `10.250.250.31:8006` +**Storage Pool**: `ospool` (CEPH/zfs — all VM disks reside here) +**Network Bridge**: `vmbr0` with VLAN tag `50` on all VMs +**QEMU Version**: 7.2.0 (primary), VM 106 on 8.1.5 + +## Virtual Machines + +### VM 100 — PFI-ANA-TRUENAS + +| Property | Value | +|---|---| +| **VMID** | 100 | +| **OS Type** | Linux (l26) | +| **CPU** | 2 sockets × 2 cores = 4 vCPU (host passthrough) | +| **Memory** | 8,196 MB | +| **Disk** | `scsi0`: 80G on ospool | +| **CDROM** | `ide2`: TrueNAS-SCALE-22.12.1.iso | +| **Network** | `net0`: virtio, MAC `9A:90:79:7A:86:87`, vmbr0, VLAN 50 | +| **Boot** | scsi0 → ide2 → net0 | +| **Startup** | Order 2, delay 120s | +| **Onboot** | No | + +**Purpose**: TrueNAS SCALE storage appliance. Provides NAS/NFS/iSCSI to the colo environment. + +--- + +### VM 101 — PFI-ANA-DC + +| Property | Value | +|---|---| +| **VMID** | 101 | +| **OS Type** | Windows 11 | +| **BIOS** | OVMF (UEFI) with TPM 2.0 | +| **Machine** | pc-q35-7.2 | +| **CPU** | 2 sockets × 6 cores = 12 vCPU (host passthrough) | +| **Memory** | 24,576 MB | +| **Disk** | `scsi0`: 240G on ospool | +| **EFI Disk** | `efidisk0`: 1M on ospool | +| **TPM** | `tpmstate0`: 4M, v2.0 on ospool | +| **CDROM** | `scsi1`: virtio-win-0.1.229.iso (VirtIO drivers) | +| **Network** | `net0`: e1000, MAC `CE:C8:D7:FE:32:40`, vmbr0, VLAN 50 | +| **Boot** | scsi0 → net0 → ide0 → scsi1 | +| **Startup** | Order 5, delay 120s | +| **Onboot** | Yes | + +**Purpose**: Windows Domain Controller for the Anaheim environment. UEFI with TPM 2.0 suggests Active Directory / Group Policy services. + +--- + +### VM 102 — PFI-ANA-Docker + +| Property | Value | +|---|---| +| **VMID** | 102 | +| **OS Type** | Linux (l26) | +| **CPU** | 2 sockets × 4 cores = 8 vCPU (x86-64-v2-AES) | +| **Memory** | 16,384 MB | +| **Disk** | `scsi0`: 250G on ospool | +| **Network** | `net0`: virtio, MAC `BA:AF:E7:E9:79:23`, vmbr0, VLAN 50 | +| **Boot** | scsi0 → net0 → scsi1 | +| **Startup** | Order 4 | +| **Onboot** | Yes | + +**Purpose**: Primary Docker host for the colo. Runs Dockge for compose management and Traefik for reverse proxy. All Docker services documented in [docker-stack.md](docker-stack.md) run here. + +--- + +### VM 103 — PFI-SlaveBot + +| Property | Value | +|---|---| +| **VMID** | 103 | +| **OS Type** | Windows 10 | +| **Machine** | pc-i440fx-7.2 | +| **CPU** | 2 sockets × 4 cores = 8 vCPU (host passthrough) | +| **Memory** | 8,192 MB | +| **Disk** | `ide0`: 256G on ospool | +| **Network** | `net0`: e1000, MAC `CE:F0:49:C9:03:70`, vmbr0, VLAN 50 | +| **Boot** | ide0 → net0 → scsi0 | +| **Startup** | Not configured | +| **Onboot** | Yes | + +**Purpose**: Windows 10 workstation/bot. Likely a task automation or RDP-accessible machine. + +--- + +### VM 104 — PFI-Mongo + +| Property | Value | +|---|---| +| **VMID** | 104 | +| **OS Type** | Linux (l26) | +| **CPU** | 2 sockets × 4 cores = 8 vCPU (host passthrough) | +| **Memory** | 8,196 MB | +| **Disk** | `scsi0`: 256G on ospool | +| **Network** | `net0`: virtio, MAC `32:57:90:B2:66:61`, vmbr0, VLAN 50 | +| **Boot** | scsi0 → ide2 → net0 | +| **Startup** | Order 3, delay 60s | +| **Onboot** | Yes | + +**Purpose**: MongoDB server. Config file contains a connection string reference: `mongodb://10.250.50.81:27017/`. + +--- + +### VM 105 — PFI-Postgres + +| Property | Value | +|---|---| +| **VMID** | 105 | +| **OS Type** | Linux (l26) | +| **CPU** | 4 sockets × 4 cores = 16 vCPU | +| **Memory** | 8,196 MB | +| **Disk** | `scsi0`: 80G on ospool | +| **CDROM** | `ide2`: debian-11.6.0-amd64-netinst.iso | +| **Network** | `net0`: virtio, MAC `C2:1F:CC:71:66:D0`, vmbr0, VLAN 50 | +| **Serial** | `serial0`: socket (IPMI/serial console) | +| **Boot** | scsi0 → ide2 → net0 | +| **Startup** | Order 3, delay 60s | +| **Onboot** | Yes | + +**Purpose**: PostgreSQL database server running Debian 11. + +--- + +### VM 106 — PFI-Tailscale + +| Property | Value | +|---|---| +| **VMID** | 106 | +| **OS Type** | Linux (l26) | +| **CPU** | 2 sockets × 4 cores = 8 vCPU (x86-64-v2-AES) | +| **Memory** | 2,048 MB | +| **Disk** | `scsi0`: 256G on ospool | +| **CDROM** | `ide2`: debian-12.2.0-amd64-netinst.iso | +| **Network** | `net0`: virtio, MAC `BC:24:11:D7:E9:52`, vmbr0, VLAN 50 | +| **Boot** | scsi0 → ide2 → net0 | +| **Startup** | Not configured | +| **Onboot** | Yes | + +**Purpose**: Tailscale VPN node for mesh connectivity. Provides the VPN tunnel endpoints that link the three PFI sites together. Running Debian 12 (newer than most other VMs). Lightweight at 2G RAM. + +--- + +### VM 107 — PFI-Pteradactyl + +| Property | Value | +|---|---| +| **VMID** | 107 | +| **OS Type** | Linux (l26) | +| **CPU** | 2 sockets × 4 cores = 8 vCPU (host passthrough) | +| **Memory** | 8,192 MB | +| **Disk** | `scsi0`: 256G on ospool | +| **CDROM** | `ide2`: debian-11.6.0-amd64-netinst.iso | +| **Network** | `net0`: virtio, MAC `CA:44:37:8A:BF:E0`, vmbr0, VLAN 50 | +| **Boot** | scsi0 → ide2 → net0 | +| **Startup** | Not configured | +| **Onboot** | Yes | + +**Purpose**: Pterodactyl game server panel. Manages game server instances. + +--- + +### VM 108 — PFI-ANA--DEV + +| Property | Value | +|---|---| +| **VMID** | 108 | +| **OS Type** | Linux (l26) | +| **CPU** | 2 sockets × 4 cores = 8 vCPU | +| **Memory** | 8,192 MB | +| **Disk** | `scsi0`: 120G on ospool | +| **CDROM** | `ide2`: debian-11.6.0-amd64-netinst.iso | +| **Network** | `net0`: virtio, MAC `F2:EF:82:2C:AF:90`, vmbr0, VLAN 50 | +| **Boot** | scsi0 → ide2 → net0 | +| **Startup** | Order 10, delay 60s | +| **Onboot** | No | + +**Purpose**: Development environment. Not set to auto-boot, starts after core infrastructure (order 10). + +--- + +### VM 110 — PFI-ANA-Webhost + +| Property | Value | +|---|---| +| **VMID** | 110 | +| **OS Type** | Linux (l26) | +| **CPU** | 4 sockets × 4 cores = 16 vCPU | +| **Memory** | 4,096 MB (balloon: 1024 MB minimum) | +| **Disk** | `scsi0`: 250G on ospool | +| **CDROM** | `ide2`: debian-11.6.0-amd64-netinst.iso | +| **Network** | `net0`: virtio, MAC `E6:F9:3A:C9:61:2A`, vmbr0, VLAN 50 | +| **Boot** | scsi0 → ide2 → net0 | +| **Startup** | Order 30, up 120s, down 120s | +| **Onboot** | Yes | + +**Purpose**: Web hosting server. Highest startup order (30) — starts last. Also has the longest graceful shutdown timeout (120s). Memory ballooning enabled for dynamic allocation. + +--- + +### VM 111 — pfi-tacticalrmm + +| Property | Value | +|---|---| +| **VMID** | 111 | +| **OS Type** | Linux (l26) | +| **CPU** | 4 sockets × 4 cores = 16 vCPU | +| **Memory** | 8,192 MB | +| **Disk** | `scsi0`: 256G on ospool | +| **Network** | `net0`: virtio, MAC `BA:FA:65:F6:46:25`, vmbr0, VLAN 50 | +| **Boot** | scsi0 → ide2 → net0 | +| **Startup** | Order 20 | +| **Onboot** | Yes | + +**Purpose**: Tactical RMM (Remote Monitoring and Management) server. Provides IT management, remote access, and monitoring capabilities. + +--- + +## Startup Order Summary + +VMs are brought up in the following order on host boot: + +| Order | VMID | Name | Delay | +|---|---|---|---| +| 2 | 100 | PFI-ANA-TRUENAS | 120s | +| 3 | 104 | PFI-Mongo | 60s | +| 3 | 105 | PFI-Postgres | 60s | +| 4 | 102 | PFI-ANA-Docker | — | +| 5 | 101 | PFI-ANA-DC | 120s | +| 10 | 108 | PFI-ANA--DEV | 60s | +| 20 | 111 | pfi-tacticalrmm | — | +| 30 | 110 | PFI-ANA-Webhost | 120s | + +VMs without a startup order (103, 106, 107) will start based on their `onboot` setting but without a specific sequencing delay. + +## Resource Summary + +| VMID | Name | vCPU | RAM (MB) | Disk | OS | +|---|---|---|---|---|---| +| 100 | PFI-ANA-TRUENAS | 4 | 8,196 | 80G | TrueNAS SCALE | +| 101 | PFI-ANA-DC | 12 | 24,576 | 240G | Windows 11 | +| 102 | PFI-ANA-Docker | 8 | 16,384 | 250G | Linux | +| 103 | PFI-SlaveBot | 8 | 8,192 | 256G | Windows 10 | +| 104 | PFI-Mongo | 8 | 8,196 | 256G | Linux | +| 105 | PFI-Postgres | 16 | 8,196 | 80G | Debian 11 | +| 106 | PFI-Tailscale | 8 | 2,048 | 256G | Debian 12 | +| 107 | PFI-Pteradactyl | 8 | 8,192 | 256G | Debian 11 | +| 108 | PFI-ANA--DEV | 8 | 8,192 | 120G | Debian 11 | +| 110 | PFI-ANA-Webhost | 16 | 4,096 | 250G | Debian 11 | +| 111 | pfi-tacticalrmm | 16 | 8,192 | 256G | Linux | +| | **Totals** | **112** | **105,348** | **2,300G** | | + +## Network Notes + +- All VMs are on **VLAN 50** via `vmbr0`. +- All VMs have **firewall enabled** on the network interface. +- Linux VMs use `virtio` network adapters; Windows VMs use `e1000`. +- The MongoDB connection string embedded in VM 104's config references IP `10.250.50.81`, suggesting VLAN 50 maps to the `10.250.50.0/24` subnet within the `10.250.0.0/16` range. diff --git a/docs/pfi/recommended-model-settings.md b/docs/pfi/recommended-model-settings.md new file mode 100644 index 0000000..aee73c5 --- /dev/null +++ b/docs/pfi/recommended-model-settings.md @@ -0,0 +1,554 @@ +# Recommended Model Inference Settings — Reference Document + +> **Source:** AIPA Knowledge Base — compiled from 5 KB reference documents. +> **Last Updated:** 2025-07-14 +> **Purpose:** Canonical reference for llama-server / llama.cpp inference parameters across all model families with KB-documented settings. + +--- + +## Table of Contents + +1. [NVIDIA Nemotron 3 Super (120B-A12B)](#1-nvidia-nemotron-3-super-120b-a12b) +2. [NVIDIA Nemotron 3 Nano (4B / 30B-A3B)](#2-nvidia-nemotron-3-nano-4b--30b-a3b) +3. [Qwen 3.5 Family (0.8B – 397B-A17B)](#3-qwen-35-family-08b--397b-a17b) +4. [Qwen3-Coder-Next (80B MoE)](#4-qwen3-coder-next-80b-moe) +5. [Google Gemma 4 Family (E2B – 31B)](#5-google-gemma-4-family-e2b--31b) +6. [Quick Reference Cards](#6-quick-reference-cards) +7. [Critical Warnings by Model](#7-critical-warnings-by-model) +8. [Models Without KB Settings](#8-models-without-kb-settings) + +--- + +## 1. NVIDIA Nemotron 3 Super (120B-A12B) + +### Model Overview + +| Property | Value | +|---|---| +| Architecture | MoE — 120B total, **12B active parameters** | +| Max Context | **1,048,576** (1M tokens) | +| Recommended Starting Context | **16K or 32K** — increase gradually | +| Reasoning Tokens | `` (ID 12), `` (ID 13) | +| Positional Embeddings | **NoPE** — YaRN NOT needed | +| Best For | Multi-agent AI, high-efficiency reasoning, coding, math | +| Performance Tier | ~GPT-5.2 / Claude Opus 4.5 level | + +### Inference Parameters + +| Parameter | General Chat / Instruction | Tool Calling | +|---|---|---| +| `temperature` | **1.0** | **0.6** | +| `top_p` | **1.0** | **0.95** | +| `min_p` | **0.01** | **0.01** | + +### Additional Settings + +| Setting | Value | Notes | +|---|---|---| +| `--seed` | **3407** | Reproducibility | +| `--prio` | **2** or **3** | Priority scheduling | +| `--special` | Required | To see reasoning tokens | +| `--verbose-prompt` | Required | To see prepended `` tokens | +| `max_new_tokens` | 32,768 – 262,144 | Up to 1M | + +### Quantization & Memory + +| Precision | Memory Required | +|---|---| +| UD-Q2_K_XL (2-bit) | ~32–36 GB | +| **UD-Q4_K_XL (4-bit)** | **~64–72 GB** | +| 8-bit | ~128 GB | +| BF16 | ~240 GB | + +### Example Command + +```bash +./llama.cpp/llama-server \ + --model Nemotron-3-Super-UD-Q4_K_XL.gguf \ + --ctx-size 16384 \ + --temp 1.0 --top-p 1.0 --min-p 0.01 \ + --seed 3407 --special \ + --flash-attn on \ + --port 8001 +``` + +--- + +## 2. NVIDIA Nemotron 3 Nano (4B / 30B-A3B) + +### Model Variants + +| Variant | Architecture | Context | Active Params | Best Fit | +|---|---|---|---|---| +| **Nano-4B** | Dense | 128K | 4B | Lightweight coding, math, agentic tasks | +| **Nano-30B-A3B** | MoE | 128K | 3B active | Best performance/size on 24GB devices | + +### Inference Parameters (Both Variants) + +| Parameter | General Chat / Instruction | Tool Calling | +|---|---|---| +| `temperature` | **1.0** | **0.6** | +| `top_p` | **1.0** | **0.95** | +| `min_p` | **0.01** | **0.01** | + +### Additional Settings + +| Setting | Value | Notes | +|---|---|---| +| `--seed` | **3407** | Reproducibility | +| `--special` | Not required | Standard chat template for Nano variants | + +### Hardware & Quantization + +#### Nano-4B + +| Precision | Memory | +|---|---| +| Q8_0 (8-bit, recommended) | ~3 GB | +| 4-bit | ~5 GB | + +#### Nano-30B-A3B + +| Precision | Memory | +|---|---| +| **UD-Q4_K_XL (4-bit, recommended)** | **~24 GB** | +| 8-bit | ~36 GB | + +### Example Commands + +```bash +# Nano-4B (8-bit) +./llama.cpp/llama-server \ + -hf unsloth/Nemotron-3-Nano-4B-GGUF:Q8_0 \ + --ctx-size 16384 \ + --temp 1.0 --top-p 1.0 --min-p 0.01 \ + --seed 3407 --flash-attn on --port 8001 + +# Nano-30B-A3B (4-bit) +./llama.cpp/llama-server \ + -hf unsloth/Nemotron-3-Nano-30B-A3B-GGUF:UD-Q4_K_XL \ + --ctx-size 16384 \ + --temp 1.0 --top-p 1.0 --min-p 0.01 \ + --seed 3407 --flash-attn on --port 8001 +``` + +--- + +## 3. Qwen 3.5 Family (0.8B – 397B-A17B) + +### Model Variants + +| Variant | Architecture | Context | Languages | Best Fit | +|---|---|---|---|---| +| **0.8B** | Dense | 256K | 201 | Smallest edge inference | +| **2B** | Dense | 256K | 201 | Small device inference | +| **4B** | Dense | 256K | 201 | Lightweight local use | +| **9B** | Dense | 256K | 201 | Capable small model | +| **27B** | Dense | 256K | 201 | Slightly more accurate than 35B-A3B; fits 18GB | +| **35B-A3B** | MoE (3B active) | 256K | 201 | Best speed/quality tradeoff; fits 22GB | +| **122B-A10B** | MoE (10B active) | 256K | 201 | High quality; needs ~70GB (4-bit) | +| **397B-A17B** | MoE (17B active) | 256K (extendable to 1M via YaRN) | 201 | Top-tier performance | + +### 27B vs 35B-A3B Decision + +- **27B** — Choose for slightly more accurate results when you can't fit a larger model. +- **35B-A3B** — Choose for much faster inference. MoE with only 3B active parameters. + +### Hardware Requirements + +| Variant | 3-bit | 4-bit | 6-bit | 8-bit | BF16 | +|---|---|---|---|---|---| +| **0.8B / 2B** | 3 GB | 3.5 GB | 5 GB | 7.5 GB | 9 GB | +| **4B** | 4.5 GB | 5.5 GB | 7 GB | 10 GB | 14 GB | +| **9B** | 5.5 GB | 6.5 GB | 9 GB | 13 GB | 19 GB | +| **27B** | 14 GB | 17 GB | 24 GB | 30 GB | 54 GB | +| **35B-A3B** | 17 GB | 22 GB | 30 GB | 38 GB | 70 GB | +| **122B-A10B** | 60 GB | 70 GB | 106 GB | 132 GB | 245 GB | +| **397B-A17B** | 180 GB | 214 GB | 340 GB | 512 GB | 810 GB | + +### Inference Parameters + +#### Thinking Mode + +| Parameter | General Tasks | Precise Coding (e.g. WebDev) | +|---|---|---| +| `temperature` | **1.0** | **0.6** | +| `top_p` | **0.95** | **0.95** | +| `top_k` | **20** | **20** | +| `min_p` | **0.0** | **0.0** | +| `presence_penalty` | **1.5** | **0.0** | +| `repetition_penalty` | **1.0** (disabled) | **1.0** (disabled) | + +#### Non-Thinking (Instruct) Mode + +| Parameter | General Tasks | Reasoning Tasks | +|---|---|---| +| `temperature` | **0.7** | **1.0** | +| `top_p` | **0.8** | **0.95** | +| `top_k` | **20** | **20** | +| `min_p` | **0.0** | **0.0** | +| `presence_penalty` | **1.5** | **1.5** | +| `repetition_penalty` | **1.0** (disabled) | **1.0** (disabled) | + +### Thinking Mode Control + +Enable thinking: +```bash +--chat-template-kwargs '{"enable_thinking":true}' +``` + +Disable thinking: +```bash +--chat-template-kwargs '{"enable_thinking":false}' +``` + +#### Default Thinking Behavior by Variant + +| Variant | Thinking Default | +|---|---| +| **0.8B, 2B, 4B, 9B** (Small) | **Disabled** — must explicitly enable | +| **27B, 35B-A3B, 122B-A10B, 397B-A17B** | **Enabled** — must explicitly disable if unwanted | + +### Context Settings + +| Setting | Value | +|---|---| +| Max context window | **262,144** (256K) | +| Context extension | Up to **1M** via YaRN | +| Recommended starting context | **16,384** (16K) for responsiveness | +| Adequate output length | **32,768** tokens | + +### Quantization Notes + +- All GGUFs use **Unsloth Dynamic 2.0** quantization — important layers upcasted to 8 or 16-bit even in 4-bit. +- Recommended starting point: **Dynamic 4-bit** (`UD-Q4_K_XL`). +- Minimum recommended: **Dynamic 2-bit** (`UD-Q2_K_XL`). + +### Example Commands + +```bash +# 35B-A3B — Thinking Mode (General) +./llama.cpp/llama-server \ + -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \ + --ctx-size 16384 \ + --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 \ + --chat-template-kwargs '{"enable_thinking":true}' \ + --flash-attn on --port 8001 + +# 9B — Thinking Enabled (small models default to disabled) +./llama.cpp/llama-server \ + -hf unsloth/Qwen3.5-9B-GGUF:UD-Q4_K_XL \ + --ctx-size 16384 \ + --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 \ + --chat-template-kwargs '{"enable_thinking":true}' \ + --flash-attn on --port 8001 +``` + +--- + +## 4. Qwen3-Coder-Next (80B MoE) + +### Model Overview + +| Property | Value | +|---|---| +| Architecture | MoE — 80B total, **3B active parameters** | +| Max Context | **262,144** (256K) | +| Recommended Context | **32,768** for less memory use | +| Thinking Mode | **Non-reasoning only** — no `` blocks | +| Best For | Fast agentic coding, long-horizon reasoning, complex tool use | +| Performance Tier | Comparable to models with 10–20× more active parameters | + +### Inference Parameters + +| Parameter | Value | Notes | +|---|---|---| +| `temperature` | **1.0** | | +| `top_p` | **0.95** | | +| `top_k` | **40** | Note: higher than Qwen3.5 general | +| `min_p` | **0.01** | llama.cpp default is 0.05 — override to 0.01 | +| `repetition_penalty` | **1.0** (disabled) | Only increase if you see looping | + +### Quick Reference One-Liner + +``` +temperature=1.0, top_p=0.95, top_k=40, min_p=0.01, repetition_penalty=1.0 +``` + +### Hardware Requirements + +| Precision | Memory Required | +|---|---| +| 3-bit (UD-IQ3_XXS) | ~34 GB | +| **4-bit (UD-Q4_K_XL)** | **~46 GB** | +| 8-bit | ~85 GB | +| BF16 | ~160 GB | + +### Key Differences from Qwen3.5 General Models + +- **No thinking mode** — this is a non-reasoning model; ultra-quick code responses +- **Higher `top_k`** (40 vs. 20) — broader sampling for creative code generation +- `enable_thinking` flag is not applicable + +### Example Commands + +```bash +# llama-server deployment +./llama.cpp/llama-server \ + --model Qwen3-Coder-Next-UD-Q4_K_XL.gguf \ + --alias "unsloth/Qwen3-Coder-Next" \ + --seed 3407 \ + --temp 1.0 --top-p 0.95 --min-p 0.01 --top-k 40 \ + --ctx-size 32768 \ + --flash-attn on --port 8001 +``` + +### vLLM FP8 Dynamic (GPU Premium) + +```bash +CUDA_VISIBLE_DEVICES='0,1,2,3' vllm serve unsloth/Qwen3-Coder-Next-FP8-Dynamic \ + --served-model-name unsloth/Qwen3-Coder-Next \ + --tensor-parallel-size 4 \ + --tool-call-parser qwen3_coder \ + --enable-auto-tool-choice \ + --dtype bfloat16 --seed 3407 \ + --max-model-len 200000 \ + --gpu-memory-utilization 0.93 \ + --port 8001 +``` + +--- + +## 5. Google Gemma 4 Family (E2B – 31B) + +### Model Variants + +| Variant | Architecture | Context | Modalities | Best Fit | +|---|---|---|---|---| +| **E2B** | Dense + PLE | 128K | Text, Image, Audio | Phone / edge, ASR, speech translation | +| **E4B** | Dense + PLE | 128K | Text, Image, Audio | Laptops, fast local multimodal | +| **26B-A4B** | MoE (4B active) | 256K | Text, Image | Best speed/quality tradeoff | +| **31B** | Dense | 256K | Text, Image | Strongest performance | + +### Inference Parameters (All Variants) + +These are Google's default Gemma 4 parameters: + +| Parameter | Value | +|---|---| +| `temperature` | **1.0** | +| `top_p` | **0.95** | +| `top_k` | **64** | +| `repetition_penalty` | **1.0** (disabled — only increase if looping) | +| End-of-sentence token | `` | + +### Context Length + +- **E2B / E4B**: max **128K** +- **26B-A4B / 31B**: max **256K** +- **Practical tip**: Start with **32K** for responsiveness, then increase as needed. + +### Hardware Requirements + +| Variant | 4-bit | 8-bit | BF16/FP16 | +|---|---|---|---| +| **E2B** | 4 GB | 5–8 GB | 10 GB | +| **E4B** | 5.5–6 GB | 9–12 GB | 16 GB | +| **26B-A4B** | 16–18 GB | 28–30 GB | 52 GB | +| **31B** | 17–20 GB | 34–38 GB | 62 GB | + +### Quantization Recommendations + +- **E2B / E4B** (small): prefer **Q8_0** (8-bit) for quality. +- **26B-A4B / 31B** (large): prefer **UD-Q4_K_XL** (Dynamic 4-bit) as starting point. + +### 26B-A4B vs 31B Decision + +- **26B-A4B** — Choose when RAM is limited. MoE with 4B active params = faster. Slight quality tradeoff. +- **31B** — Choose when you have ≥20 GB (4-bit) and want maximum quality. Slower inference. + +### Thinking Mode + +**Enable thinking** — add to system prompt: +``` +<|think|> +You are a careful coding assistant. Explain your answer clearly. +``` + +Model outputs: +``` +<|channel>thought +[internal reasoning] + +[final answer] +``` + +**Disable thinking** via llama-server: +```bash +--chat-template-kwargs '{"enable_thinking":false}' +``` + +> **Multi-turn rule:** Only keep the final visible answer in chat history. Do NOT feed prior thought blocks back. + +### Multimodal Settings + +- **Images/Audio** go **before text** in prompts. +- **Video**: pass frames first, then instruction. +- **Audio** is only on E2B and E4B. Max audio: 30s. Max video: 60s (1 fps). + +#### Visual Token Budgets + +| Budget | Use Case | +|---|---| +| 70 / 140 | Classification, captioning, fast video | +| 280 / 560 | General multimodal chat, charts, UI | +| 1120 | OCR, document parsing, handwriting | + +### Example Commands + +```bash +# 26B-A4B (Dynamic 4-bit) +./llama.cpp/llama-server \ + -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL \ + --temp 1.0 --top-p 0.95 --top-k 64 \ + --ctx-size 32768 --flash-attn on --port 8001 + +# 31B (Dynamic 4-bit) with thinking +./llama.cpp/llama-server \ + -hf unsloth/gemma-4-31B-it-GGUF:UD-Q4_K_XL \ + --temp 1.0 --top-p 0.95 --top-k 64 \ + --ctx-size 32768 \ + --chat-template-kwargs '{"enable_thinking":true}' \ + --flash-attn on --port 8001 +``` + +--- + +## 6. Quick Reference Cards + +### Temperature by Model & Use Case + +| Model Family | General / Chat | Coding / Precise | Tool Calling | Non-Thinking | +|---|---|---|---|---| +| **Nemotron 3 Super** | 1.0 | — | 0.6 | — | +| **Nemotron 3 Nano** | 1.0 | — | 0.6 | — | +| **Qwen 3.5** (thinking) | 1.0 | 0.6 | — | — | +| **Qwen 3.5** (non-thinking) | — | — | — | 0.7 (general) / 1.0 (reasoning) | +| **Qwen3-Coder-Next** | 1.0 | 1.0 | — | — | +| **Gemma 4** | 1.0 | 1.0 | — | — | + +### Top-P by Model & Use Case + +| Model Family | General / Chat | Coding / Precise | Tool Calling | Non-Thinking | +|---|---|---|---|---| +| **Nemotron 3 Super** | 1.0 | — | 0.95 | — | +| **Nemotron 3 Nano** | 1.0 | — | 0.95 | — | +| **Qwen 3.5** (thinking) | 0.95 | 0.95 | — | — | +| **Qwen 3.5** (non-thinking) | — | — | — | 0.8 (general) / 0.95 (reasoning) | +| **Qwen3-Coder-Next** | 0.95 | 0.95 | — | — | +| **Gemma 4** | 0.95 | 0.95 | — | — | + +### Top-K by Model + +| Model Family | Top-K | +|---|---| +| **Nemotron 3** | Default (not specified) | +| **Qwen 3.5** | **20** | +| **Qwen3-Coder-Next** | **40** | +| **Gemma 4** | **64** | + +### Min-P by Model + +| Model Family | Min-P | +|---|---| +| **Nemotron 3 (all)** | **0.01** | +| **Qwen 3.5** | **0.0** | +| **Qwen3-Coder-Next** | **0.01** | +| **Gemma 4** | Default (not specified) | + +### Presence Penalty by Model + +| Model Family | General | Coding | Reasoning | +|---|---|---|---| +| **Nemotron 3** | Default | Default | Default | +| **Qwen 3.5** (thinking) | **1.5** | **0.0** | **1.5** | +| **Qwen 3.5** (non-thinking) | **1.5** | — | **1.5** | +| **Qwen3-Coder-Next** | Default | Default | Default | +| **Gemma 4** | Default | Default | Default | + +### Repetition Penalty by Model + +| Model Family | Value | Notes | +|---|---|---| +| **Nemotron 3** | Default | — | +| **Qwen 3.5** | **1.0** (disabled) | — | +| **Qwen3-Coder-Next** | **1.0** (disabled) | Only increase if looping | +| **Gemma 4** | **1.0** (disabled) | Only increase if looping | + +### Seed Values + +| Model Family | Recommended Seed | +|---|---| +| **Nemotron 3 (all)** | **3407** | +| **Qwen 3.5** | 3407 (optional) | +| **Qwen3-Coder-Next** | **3407** | +| **Gemma 4** | Default | + +--- + +## 7. Critical Warnings by Model + +### Nemotron 3 Super +- ⚠️ Do NOT attempt 1M context on first run — increase gradually from 16K/32K. +- ⚠️ Setting context to 1M may trigger **CUDA OOM and crash**. +- ⚠️ Router-layer fine-tuning **disabled by default** in Unsloth for MoE models. + +### Qwen 3.5 +- ⚠️ **No Qwen3.5 GGUF works in Ollama** due to separate mmproj vision files. Use llama.cpp-compatible backends. +- ⚠️ `presence_penalty` above 0.0 may cause **slight performance decrease**. +- ⚠️ If getting gibberish, check context length or add `--cache-type-k bf16 --cache-type-v bf16`. +- ⚠️ Redownload older GGUFs — all updated with improved quantization and tool-calling template fixes. + +### Qwen3-Coder-Next +- ⚠️ **Update llama.cpp** — a previous bug in `vectorized key_gdiff` caused looping/output issues. +- ⚠️ **No Ollama support** — use llama.cpp-compatible backends. +- ⚠️ If context length too low, may see `exceeds the available context size` errors. +- ⚠️ Tool-calling improved after llama.cpp parsing fixes (Feb 19 update) — use recent version. + +### Gemma 4 +- ⚠️ **Do NOT use CUDA 13.2 runtime** for any GGUF — causes poor outputs. +- ⚠️ Use `llama-server` (not `llama-cli`) for thinking control — more reliable. +- ⚠️ Multi-turn: **only keep the final visible answer** in chat history. Do NOT feed prior thought blocks back. + +--- + +## 8. Models Without KB Settings + +The following model families are deployed in the Infrastructure-PFI environment but **do not have KB-documented inference parameters**. Settings for these models use general best practices or vendor defaults: + +| Model | Notes | +|---|---| +| **DeepSeek R1 0528** | No KB doc — use general MoE defaults | +| **Mistral Small 3.1** | No KB doc — use vendor defaults | +| **GLM-4.7-Flash** | No KB doc — use vendor defaults | +| **GLM Steam 106B-A12B** | No KB doc — use general MoE defaults | +| **Granite 4.0 Micro** | No KB doc — use IBM defaults | +| **Kimi K2** | No KB doc — use general MoE defaults | +| **Skyfall R1 31B v4a** | No KB doc — use general defaults | +| **Hermes 4 14B** | No KB doc (download pending) — use vendor defaults | + +--- + +## KB Source Documents + +| Document | Path in KB | +|---|---| +| Nemotron 3 Super Running Parameters | `reference/nemotron-3-super-running-parameters.md` | +| Nemotron 3 Nano Running Parameters | `reference/nemotron-3-nano-running-parameters.md` | +| Qwen 3.5 Running Parameters | `reference/qwen3.5-running-parameters.md` | +| Qwen3-Coder-Next Running Parameters | `reference/qwen3-coder-next-running-parameters.md` | +| Gemma 4 Running Parameters | `reference/gemma-4-running-parameters.md` | + +--- + +*Document generated by Linus (Systems Architect) from AIPA Knowledge Base content curated by Atlas.* diff --git a/docs/pfi/vm-102-matrix-appservice.md b/docs/pfi/vm-102-matrix-appservice.md new file mode 100644 index 0000000..b97c3cf --- /dev/null +++ b/docs/pfi/vm-102-matrix-appservice.md @@ -0,0 +1,261 @@ +--- +id: vm-102-matrix-appservice +title: "VM 102 — Matrix Appservice Configuration" +summary: "AIPA Matrix Application Service bridge setup for VM 102. Covers appservice registration, agent virtual users, room routing, env vars, operational notes, and troubleshooting." +tags: ["infrastructure", "pfi", "pfi-ana", "matrix", "docker", "vm-102", "appservice", "bridge", "aipa", "deployment", "integration"] +keywords: ["matrix.pfi.local", "appservice", "as_token", "hs_token", "aipa-bridge", "matrix_bridge.py", "8009", "send_notification", "virtual-users", "room-routing", "element"] +links: + - "[VM 102 — Matrix Synapse Deployment](vm-102-matrix-synapse.md)" + - "[Matrix Bridge — Application Service Integration](../../../KB/projects/aipa/matrix-bridge.md)" + - "[Docker Stack Conventions](docker-stack.md)" +created: 2026-04-11T00:00:00+00:00 +modified: 2026-04-11T00:00:00+00:00 +path: docs/pfi-ana/vm-102-matrix-appservice.md +--- + +# VM 102 — Matrix Appservice Configuration + +## Overview + +The AIPA Matrix bridge registers with Synapse as a Matrix Application Service. This means: +- Synapse **pushes** all relevant events to the bridge (no polling) +- The bridge manages **virtual users** for each agent — no real accounts needed +- The bridge authenticates to Synapse with a shared `as_token` +- Synapse authenticates its pushes to the bridge with a shared `hs_token` + +> Prerequisite: [VM 102 — Matrix Synapse Deployment](vm-102-matrix-synapse.md) must be complete. + +--- + +## Application Service Model + +``` +Synapse ──── push events ────► AIPA Bridge (port 8009) + │ + routes to agent + │ + posts response ◄──── Synapse ◄──── Element ◄──── User +``` + +### Agent Virtual Users + +| Agent | Matrix ID | Display Name | +|----------|----------------------------|--------------| +| Atlas | `@atlas:matrix.pfi.local` | Atlas | +| Linus | `@linus:matrix.pfi.local` | Linus | +| Hermione | `@hermione:matrix.pfi.local` | Hermione | + +These are **virtual** — managed entirely by the bridge. Do not register them as real Synapse accounts. + +--- + +## Step 1 — Generate Tokens + +Two random tokens are needed: + +```bash +python3 -c "import secrets; print(secrets.token_hex(32))" # as_token (bridge → Synapse) +python3 -c "import secrets; print(secrets.token_hex(32))" # hs_token (Synapse → bridge) +``` + +--- + +## Step 2 — Create Appservice Registration File + +File: `/opt/docker/data/synapse/aipa_appservice.yaml` + +```yaml +id: aipa-bridge +url: http://:8009 # IP/hostname the bridge is reachable from Synapse container +as_token: "" +hs_token: "" +sender_localpart: aipa-bot # @aipa-bot:matrix.pfi.local (unused fallback sender) +namespaces: + users: + - exclusive: true + regex: "@(atlas|linus|hermione):.*" + rooms: [] + aliases: [] +rate_limited: false +``` + +- **`exclusive: true`** — only the bridge can act as those users; no one can register `@atlas` as a real account. +- **`url`** — must be reachable from inside the Synapse container. If the bridge runs on the VM host, use the host IP or Docker gateway IP (e.g., `172.17.0.1`). + +### Restart Synapse to Load + +```bash +cd /opt/docker/compose/synapse && docker compose restart synapse +``` + +--- + +## Step 3 — Configure AIPA Environment + +### env.sh additions + +```bash +# ── Matrix bridge ────────────────────────────────────────────────────── +export MATRIX_HOMESERVER_URL="http://localhost:8008" +export MATRIX_SERVER_NAME="matrix.pfi.local" +export MATRIX_AS_TOKEN="" +export MATRIX_HS_TOKEN="" +# Optional: room to send agent notifications when no channel is specified +# export MATRIX_DEFAULT_ROOM="!roomid:matrix.pfi.local" +``` + +### providers.yaml matrix section + +```yaml +matrix: + homeserver_url: "${MATRIX_HOMESERVER_URL}" + server_name: "${MATRIX_SERVER_NAME}" + as_token: "${MATRIX_AS_TOKEN}" + hs_token: "${MATRIX_HS_TOKEN}" + bridge_host: "0.0.0.0" + bridge_port: 8009 + default_notification_room: "${MATRIX_DEFAULT_ROOM}" + agents: + atlas: + display_name: "Atlas" + avatar_url: "" + linus: + display_name: "Linus" + avatar_url: "" + hermione: + display_name: "Hermione" + avatar_url: "" +``` + +--- + +## Step 4 — Start the Bridge + +```bash +source .venv/bin/activate && source env.sh +python -m core.matrix_bridge +``` + +Options: + +``` +python -m core.matrix_bridge --host 0.0.0.0 --port 8009 +python -m core.matrix_bridge --no-profile-setup # skip display name / avatar init +``` + +On first start, the bridge: +1. Registers display names for each agent user +2. Starts the appservice HTTP server on port 8009 +3. Listens for Matrix transactions from Synapse +4. Accepts invitations to rooms +5. Begins routing messages + +--- + +## Step 5 — First Use + +1. Open Element at `http://10.250.50.70:8080` +2. Log in with your admin account (server: `matrix.pfi.local`) +3. Open a DM with `@atlas:matrix.pfi.local` +4. Send a message — the bridge routes it to Atlas and posts the response + +To start a room with a specific agent, invite them: +- New room → Invite `@linus:matrix.pfi.local` → Linus joins automatically +- All messages in that room go to Linus + +--- + +## Room Routing + +1. User invites an agent user to a room (or opens a DM) +2. Bridge receives the `m.room.member` invite event, agent auto-joins +3. Room is permanently mapped to that agent in `sessions/matrix_rooms.json` +4. All subsequent messages in that room are routed to the mapped agent +5. If multiple agent users are in the same room, the first invite wins for session purposes; + subsequent agents each get their own session (multi-agent collaboration rooms) + +--- + +## Operational Notes + +### Room Session Persistence + +The bridge persists the room→agent→session mapping to `sessions/matrix_rooms.json` +under `AIPA_ROOT`. If the bridge restarts, existing rooms continue their sessions. + +To reset a room's conversation history: +- Ask the agent `/reset`, or +- Delete the entry from `matrix_rooms.json` and restart the bridge. + +### Formatting + +The bridge converts markdown in agent responses to Matrix HTML +(`format: org.matrix.custom.html`). Code blocks, bold, and inline code are +rendered correctly in Element. + +### Typing Indicators + +The bridge sends a typing indicator (`m.typing`) while the agent is processing, +so users see the animated dots while waiting. + +### Proactive Notifications + +Agents can push messages to Matrix rooms via `send_notification`: + +``` +send_notification( + content="KB rebuild complete — 1,247 chunks indexed.", + channel="!roomid:matrix.pfi.local" +) +``` + +- If `channel` is set, it must be a Matrix room ID (`!roomid:server`) +- If `channel` is omitted, the notification goes to `MATRIX_DEFAULT_ROOM` if configured + +### Agent Users Always Show as Offline + +Expected behavior — virtual users don't have presence. This is normal for appservice users. + +--- + +## Troubleshooting + +| Symptom | Likely Cause | Fix | +|---------|-------------|-----| +| Bridge starts but Synapse doesn't push events | Appservice URL wrong in registration YAML | Verify `url:` is reachable from the Synapse container; check `docker inspect synapse` network | +| Agent joins room but doesn't respond | `hs_token` mismatch | Verify token in `aipa_appservice.yaml` matches `MATRIX_HS_TOKEN` env var | +| 401 errors from Synapse | `as_token` mismatch | Verify token in `aipa_appservice.yaml` matches `MATRIX_AS_TOKEN` env var | +| Agent user shows as offline always | Expected — virtual users don't have presence | Normal for appservice users; no fix needed | +| Messages loop (agent replies to itself) | Bridge not filtering its own messages | Check `SENDER_LOCALPART` filter in bridge; bridge ignores all managed agent users as senders | +| Element can't connect to homeserver | Wrong `base_url` in element-config.json | Must be the IP/hostname Element's browser can reach, not the Docker container name | + +--- + +## Dependencies + +| Dependency | Version | Purpose | +|------------|---------|---------| +| `markdown` | any | Markdown→HTML rendering for Matrix responses | +| `fastapi` | any | Appservice HTTP server | +| `httpx` | any | Client-Server API calls to Synapse | + +Install: `pip install markdown` (or `pip install -r requirements.txt`) + +--- + +## Security Notes + +- The `as_token` and `hs_token` are secrets equivalent to admin credentials. Store + them in `env.sh` (gitignored) and never commit them. +- The bridge runs with `MATRIX_HS_TOKEN` to authenticate Synapse's push requests. Any + request without this token in the `Authorization` header is rejected (403). +- Restrict port 8009 to internal access only (firewall or bind to `127.0.0.1` if bridge and Synapse are on the same host). + +--- + +## Sources + +- Source deployment guide: `projects/matrix/matrix-deployment.md` (2026-04-11) +- AIPA bridge code: `core/matrix_bridge.py` +- AIPA bridge KB entry: `KB/projects/aipa/matrix-bridge.md` +- Matrix Synapse appservice docs: https://element-hq.github.io/synapse/latest/application_services.html diff --git a/docs/pfi/vm-102-matrix-synapse.md b/docs/pfi/vm-102-matrix-synapse.md new file mode 100644 index 0000000..f715cb5 --- /dev/null +++ b/docs/pfi/vm-102-matrix-synapse.md @@ -0,0 +1,340 @@ +--- +id: vm-102-matrix-synapse +title: "VM 102 — Matrix Synapse Deployment" +summary: "Docker deployment of Synapse homeserver with PostgreSQL and Element Web on VM 102 (PFI-ANA-Docker). Covers compose file, configuration, storage paths, and admin setup." +tags: ["infrastructure", "pfi", "pfi-ana", "matrix", "docker", "vm-102", "synapse", "self-hosted", "deployment"] +keywords: ["matrix.pfi.local", "synapse", "element-web", "10.250.50.70", "VM-102", "PFI-ANA-Docker", "postgres:16", "appservice", "AIPA"] +links: + - "[Matrix Protocol Reference](../Infrastructure-PFI-Project-Index.md)" + - "[VM-102 Proxmox Config](../../../configs/pfi-ana/proxmox/vm-102.conf)" + - "[Docker Stack Conventions](docker-stack.md)" +created: 2026-04-11T00:00:00+00:00 +modified: 2026-04-11T00:00:00+00:00 +path: docs/pfi-ana/vm-102-matrix-synapse.md +--- + +# VM 102 — Matrix Synapse Deployment + +## Overview + +Matrix (Synapse) is deployed on VM 102 (PFI-ANA-Docker, `10.250.50.70`) as the primary +human-to-agent communication channel for AIPA. The stack consists of: + +- **Synapse** — Matrix homeserver (event routing, auth, persistence) +- **PostgreSQL 16** — Synapse database backend +- **Element Web** — Web client for users + +Federation is disabled (internal-only deployment). Registration is disabled (admin-created accounts only). + +--- + +## Architecture + +``` +User (Element client) + │ m.room.message events + ▼ +┌──────────────────────┐ +│ Synapse │ Matrix homeserver — event routing, auth, persistence +│ (homeserver) │ Port 8008 (Client-Server API) +└──────────┬───────────┘ + │ Appservice push PUT /_matrix/app/v1/transactions/{txnId} + ▼ +┌──────────────────────────────────────────┐ +│ AIPA Matrix Bridge (core/matrix_bridge.py) │ +│ Port 8009 │ +└──────────────────────────────────────────┘ +``` + +> Full appservice bridge details: [VM 102 — Matrix Appservice Configuration](vm-102-matrix-appservice.md) + +--- + +## Components + +| Component | Image | Port | Purpose | +|-----------------|--------------------------------|-------|-----------------------------------| +| `synapse` | `matrixdotorg/synapse:latest` | 8008 | Matrix homeserver (Client-Server API) | +| `synapse-db` | `postgres:16` | — | Synapse database | +| `element-web` | `vectorim/element-web:latest` | 8080 | Web client for users | + +--- + +## Storage Paths + +| Host Path | Purpose | +|------------------------------------|-----------------------------------------| +| `/opt/docker/compose/synapse/` | Compose file (managed by Dockge) | +| `/opt/docker/conf/synapse/` | Configuration files | +| `/opt/docker/data/synapse/` | Synapse data (`homeserver.yaml`, signing keys, appservice reg) | +| `/opt/docker/conf/synapse/element-config.json` | Element Web client config | + +> Follows the VM 102 convention: compose in `/opt/docker/compose//`, config in `/opt/docker/conf//`. + +### Volumes + +| Volume / Path | Container Path | Purpose | +|--------------------------------|-------------------------------|-----------------------------| +| `/opt/docker/data/synapse` | `/data` | Synapse config, signing keys, media store | +| `synapse-db-data` (named vol) | `/var/lib/postgresql/data` | PostgreSQL persistent data | + +--- + +## Step 1 — Create Directory Structure + +```bash +mkdir -p /opt/docker/data/synapse +mkdir -p /opt/docker/conf/synapse +mkdir -p /opt/docker/compose/synapse +``` + +--- + +## Step 2 — Generate Synapse Config + +```bash +docker run --rm \ + -v /opt/docker/data/synapse:/data \ + -e SYNAPSE_SERVER_NAME=matrix.pfi.local \ + -e SYNAPSE_REPORT_STATS=no \ + matrixdotorg/synapse:latest generate +``` + +This writes `/opt/docker/data/synapse/homeserver.yaml` and +`/opt/docker/data/synapse/matrix.pfi.local.signing.key`. + +**Do not modify `server_name` after generation — it is permanent.** + +--- + +## Step 3 — Edit homeserver.yaml + +Open `/opt/docker/data/synapse/homeserver.yaml` and apply: + +```yaml +# Use PostgreSQL instead of SQLite (required for production) +database: + name: psycopg2 + args: + user: synapse + password: synapse_db_password # match POSTGRES_PASSWORD in compose + database: synapse + host: synapse-db + cp_min: 5 + cp_max: 10 + +# Disable open registration — accounts are created by admin only +enable_registration: false + +# Disable federation (internal deployment only) +federation_domain_whitelist: [] + +# Allow the application service to be registered (add AFTER generating the AS file) +app_service_config_files: + - /data/aipa_appservice.yaml +``` + +--- + +## Step 4 — Docker Compose + +File: `/opt/docker/compose/synapse/docker-compose.yml` + +```yaml +--- +# ============================================================================= +# Synapse Matrix Homeserver — AIPA internal deployment on VM 102 +# ============================================================================= +# +# Conventions: +# - Config: /opt/docker/conf/synapse/ +# - Data: /opt/docker/data/synapse/ (bind mount) + synapse-db-data (named vol) +# - Compose: /opt/docker/compose/synapse/ +# - Network: synapse-net (dedicated, not on traefik-net) +# +# Notes: +# - Federation disabled (internal only) +# - Port 8448 (federation) commented out +# - Resource limits set for VM 102 (8 vCPU, 16 GB RAM) + +services: + synapse-db: + image: postgres:16 + container_name: synapse-db + restart: unless-stopped + environment: + POSTGRES_USER: synapse + POSTGRES_PASSWORD: synapse_db_password + POSTGRES_DB: synapse + POSTGRES_INITDB_ARGS: "--encoding=UTF-8 --lc-collate=C --lc-ctype=C" + volumes: + - synapse-db-data:/var/lib/postgresql/data + networks: + - synapse-net + healthcheck: + test: ["CMD-SHELL", "pg_isready -U synapse"] + interval: 10s + timeout: 5s + retries: 5 + + synapse: + image: matrixdotorg/synapse:latest + container_name: synapse + restart: unless-stopped + depends_on: + synapse-db: + condition: service_healthy + ports: + - "8008:8008" # Client-Server API (HTTP) + # - "8448:8448" # Server-Server API (federation) — disabled for internal use + volumes: + - /opt/docker/data/synapse:/data + networks: + - synapse-net + deploy: + resources: + limits: + memory: 1G + cpus: "2.0" + reservations: + memory: 256M + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:8008/health || exit 1"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 30s + + element-web: + image: vectorim/element-web:latest + container_name: element-web + restart: unless-stopped + volumes: + - /opt/docker/conf/synapse/element-config.json:/app/config.json:ro + ports: + - "8080:80" + networks: + - synapse-net + +volumes: + synapse-db-data: + +networks: + synapse-net: + name: synapse-net +``` + +### Start + +```bash +cd /opt/docker/compose/synapse +docker compose up -d +``` + +### Verify + +```bash +docker compose ps +curl http://localhost:8008/health +``` + +--- + +## Step 5 — Element Web Configuration + +File: `/opt/docker/conf/synapse/element-config.json` + +```json +{ + "default_server_config": { + "m.homeserver": { + "base_url": "http://10.250.50.70:8008", + "server_name": "matrix.pfi.local" + } + }, + "brand": "AIPA", + "default_theme": "dark", + "disable_guests": true, + "disable_login_language_selector": true +} +``` + +--- + +## Step 6 — Create Admin User + +```bash +docker exec -it synapse register_new_matrix_user \ + -u admin \ + -p 'yourpassword' \ + -a \ + http://localhost:8008 +``` + +> The `-a` flag makes the user an admin. Omit for regular users. + +**Agent users** (`@atlas`, `@linus`, `@hermione`) are **virtual** — managed by the +appservice. Do **not** register them as real accounts. + +--- + +## Port Allocation + +| Port | Service | Purpose | Protocol | +|------|---------------|----------------------------|----------| +| 8008 | Synapse | Client-Server API | HTTP | +| 8080 | Element Web | Web client | HTTP | +| 8009 | AIPA Bridge | Appservice endpoint | HTTP | + +> See [docker-stack.md](docker-stack.md) for the full VM 102 port allocation table. + +--- + +## Network + +This deployment uses a **dedicated `synapse-net` network** (not `traefik-net`), because: + +- Synapse is accessed directly by IP (no public domain routing needed) +- The AIPA bridge connects to Synapse at `http://localhost:8008` from the host +- Element Web connects at the VM IP:8008 from the browser + +If TLS/reverse proxy is added later, join `traefik-net` and add Traefik labels. + +--- + +## Security Notes + +- Synapse is exposed on port 8008 (HTTP). For any externally accessible deployment, + put it behind a TLS-terminating reverse proxy (Traefik/nginx) and restrict 8009 + to internal access only. +- Registration is disabled (`enable_registration: false`) — accounts created by admin only. +- Federation is disabled (`federation_domain_whitelist: []`) — internal use only. +- The `as_token` and `hs_token` in the appservice registration are secrets equivalent to + admin credentials. Store in `env.sh` (gitignored), never commit. + +--- + +## Troubleshooting + +| Symptom | Likely Cause | Fix | +|---------|-------------|-----| +| Bridge starts but Synapse doesn't push events | Appservice URL wrong in registration YAML | Verify `url:` is reachable from the Synapse container; check `docker inspect synapse` network | +| 401 errors from Synapse | `as_token` mismatch | Verify token in `aipa_appservice.yaml` matches `MATRIX_AS_TOKEN` env var | +| Element can't connect to homeserver | Wrong `base_url` in element-config.json | Must be the IP/hostname Element's browser can reach, not the Docker container name | +| Synapse won't start | Database connection failure | Verify `synapse-db` is healthy first; check password matches in `homeserver.yaml` and compose | + +--- + +## Next Steps + +After Synapse is running and healthy, configure the AIPA appservice bridge: +→ [VM 102 — Matrix Appservice Configuration](vm-102-matrix-appservice.md) + +--- + +## Sources + +- Source deployment guide: `projects/matrix/matrix-deployment.md` (2026-04-11) +- VM 102 Proxmox config: `configs/pfi-ana/proxmox/vm-102.conf` +- Docker Stack conventions: `docs/pfi-ana/docker-stack.md` +- Matrix Protocol Reference: `infrastructure/matrix-docker-deployment.md` (KB) diff --git a/scripts/add-host.sh b/scripts/add-host.sh new file mode 100755 index 0000000..ffe5ff0 --- /dev/null +++ b/scripts/add-host.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash +# add-host.sh — register a new server so refresh-server-info.sh picks it up. +# +# Creates servers// and writes servers//ssh-target with the +# given IP (or user@ip). The next `scripts/refresh-server-info.sh ` +# run will discover the host and pull its first system-details.txt. +# +# Usage: +# scripts/add-host.sh +# scripts/add-host.sh --force # overwrite existing +# +# Example: +# scripts/add-host.sh la-docker 10.60.50.12 +# scripts/add-host.sh edge-01 admin@203.0.113.9 + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +SERVERS_DIR="$REPO_ROOT/servers" + +FORCE=0 +POSITIONAL=() +for arg in "$@"; do + case "$arg" in + -f|--force) FORCE=1 ;; + -h|--help) sed -n '2,14p' "$0"; exit 0 ;; + -*) echo "error: unknown flag $arg" >&2; exit 2 ;; + *) POSITIONAL+=("$arg") ;; + esac +done + +if [ "${#POSITIONAL[@]}" -ne 2 ]; then + echo "usage: $(basename "$0") [--force]" >&2 + exit 2 +fi + +name="${POSITIONAL[0]}" +target="${POSITIONAL[1]}" + +if [[ ! "$name" =~ ^[a-zA-Z0-9][a-zA-Z0-9._-]*$ ]]; then + echo "error: invalid host name '$name' (expected [a-zA-Z0-9._-])" >&2 + exit 2 +fi + +# Trivial sanity check on the target — not exhaustive, just catches typos. +if [[ -z "$target" ]] || [[ "$target" =~ [[:space:]] ]]; then + echo "error: invalid ssh target '$target'" >&2 + exit 2 +fi + +host_dir="$SERVERS_DIR/$name" +ssh_target_file="$host_dir/ssh-target" + +if [ -f "$ssh_target_file" ] && [ "$FORCE" -ne 1 ]; then + existing=$(awk 'NF{print $1; exit}' "$ssh_target_file") + if [ "$existing" = "$target" ]; then + echo "host '$name' already registered with target '$target' — nothing to do" + exit 0 + fi + echo "error: $ssh_target_file already exists (currently '$existing'); pass --force to overwrite" >&2 + exit 1 +fi + +mkdir -p "$host_dir" +printf '%s\n' "$target" > "$ssh_target_file" + +printf 'registered: %s → %s\n' "$name" "$target" +printf 'next: scripts/refresh-server-info.sh %s\n' "$name" diff --git a/scripts/deploy-stack.sh b/scripts/deploy-stack.sh new file mode 100755 index 0000000..0eb7c11 --- /dev/null +++ b/scripts/deploy-stack.sh @@ -0,0 +1,257 @@ +#!/usr/bin/env bash +# deploy-stack.sh — push a local stacks-mirror dir to a server, with +# per-file diff and confirmation prompt. +# +# Layout assumed: +# stacks-mirror/// → :/opt/docker/compose// +# stacks-mirror///conf/ → :/opt/docker/conf// +# +# Secrets / runtime state are never pushed (same exclude list as +# sync-stacks.sh): .env*, acme.json, *.key/crt/pem/pfx, *.sqlite*, *.db, +# *.log*, *.pid, hub/, logs/, client_secrets.json. +# +# The script: +# 1. Runs rsync --dry-run to find which files would change. +# 2. Prints a unified diff for each changed/added file (deletions noted). +# 3. Prompts [y/N]; applies the rsync only on 'y'. +# +# Usage: +# scripts/deploy-stack.sh +# scripts/deploy-stack.sh --yes # skip prompt (use sparingly) +# scripts/deploy-stack.sh --compose # push only compose side +# scripts/deploy-stack.sh --conf # push only conf side + +set -euo pipefail + +if ! command -v rsync >/dev/null 2>&1; then + echo "error: rsync is not installed on this workstation" >&2 + echo " install it (e.g. 'sudo apt install rsync') and ensure the target host has it too" >&2 + exit 2 +fi + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +SERVERS_DIR="$REPO_ROOT/servers" +MIRROR_DIR="$REPO_ROOT/stacks-mirror" + +EXCLUDES=( + # Include .env.example / *.env.example templates before the broader + # .env* exclude — rsync processes these in order, first match wins. + --include='.env.example' + --include='*.env.example' + --exclude=.env + --exclude='.env.*' + --exclude=acme.json + --exclude=client_secrets.json + --exclude='*.pem' + --exclude='*.key' + --exclude='*.crt' + --exclude='*.pfx' + --exclude='*.sqlite' + --exclude='*.sqlite3' + --exclude='*.db' + --exclude='*.log' + --exclude='*.log.*' + --exclude='*.pid' + --exclude='hub/' + --exclude='logs/' +) + +HOST= +STACK= +ASSUME_YES=0 +DO_COMPOSE=1 +DO_CONF=1 +for a in "$@"; do + case "$a" in + --yes|-y) ASSUME_YES=1 ;; + --compose) DO_CONF=0 ;; + --conf) DO_COMPOSE=0 ;; + -h|--help) sed -n '2,22p' "$0"; exit 0 ;; + -*) echo "error: unknown flag $a" >&2; exit 2 ;; + *) + if [ -z "$HOST" ]; then HOST="$a" + elif [ -z "$STACK" ]; then STACK="$a" + else echo "error: unexpected positional '$a'" >&2; exit 2 + fi + ;; + esac +done + +[ -n "$HOST" ] || { echo "usage: $(basename "$0") " >&2; exit 2; } +[ -n "$STACK" ] || { echo "usage: $(basename "$0") " >&2; exit 2; } + +resolve_target() { + local host="$1" + local effective + effective=$(ssh -G "$host" 2>/dev/null | awk '/^hostname /{print $2; exit}') + if [ -n "$effective" ] && getent hosts "$effective" >/dev/null 2>&1; then + echo "$host"; return + fi + local fb="$SERVERS_DIR/$host/ssh-target" + if [ -f "$fb" ]; then awk 'NF{print $1; exit}' "$fb"; return; fi + echo "$host" +} + +TARGET=$(resolve_target "$HOST") +STACK_DIR="$MIRROR_DIR/$HOST/$STACK" + +[ -d "$STACK_DIR" ] || { echo "error: $STACK_DIR not found (pull with sync-stacks.sh first)" >&2; exit 2; } + +# Collect the two src/dest pairs we need to consider. +PAIRS=() # each entry: "||" +if [ "$DO_COMPOSE" -eq 1 ]; then + PAIRS+=("compose|$STACK_DIR/|$TARGET:/opt/docker/compose/$STACK/") +fi +if [ "$DO_CONF" -eq 1 ] && [ -d "$STACK_DIR/conf" ]; then + PAIRS+=("conf|$STACK_DIR/conf/|$TARGET:/opt/docker/conf/$STACK/") +fi + +[ "${#PAIRS[@]}" -gt 0 ] || { echo "nothing to deploy"; exit 0; } + +# --------- Dry-run summary: which files would change, per kind. -------- +declare -A CHANGED_FILES_BY_KIND=() # kind → newline-separated list +declare -A DELETED_FILES_BY_KIND=() +declare -A RAW_RSYNC_OUT_BY_KIND=() # kind → raw rsync itemize output +any_change=0 + +for entry in "${PAIRS[@]}"; do + IFS='|' read -r kind src dest <<<"$entry" + extra=() + [ "$kind" = compose ] && extra+=(--exclude='conf/') + + # Pre-create the remote dir. Without this, rsync against a nonexistent + # destination can fail in ways the dry-run doesn't surface cleanly. + remote_path="/opt/docker/$kind/$STACK/" + if ! ssh -o BatchMode=yes -o ConnectTimeout=10 "$TARGET" \ + "mkdir -p '$remote_path'" 2>/dev/null; then + echo "error: could not create $remote_path on $TARGET (check perms / ssh)" >&2 + exit 2 + fi + + tmp_out=$(mktemp) tmp_err=$(mktemp) + rc=0 + rsync -az --delete --dry-run \ + --out-format='%i %n' \ + "${EXCLUDES[@]}" "${extra[@]}" \ + "$src" "$dest" >"$tmp_out" 2>"$tmp_err" || rc=$? + if [ "$rc" -ne 0 ]; then + echo "error: rsync dry-run failed (exit $rc) for $src → $dest" >&2 + sed 's/^/ /' "$tmp_err" >&2 + rm -f "$tmp_out" "$tmp_err" + exit 2 + fi + mapfile -t lines < "$tmp_out" + RAW_RSYNC_OUT_BY_KIND[$kind]=$(cat "$tmp_out") + rm -f "$tmp_out" "$tmp_err" + + changed="" + deleted="" + for ln in "${lines[@]}"; do + # Itemized codes (rsync uses '<' for push, '>' for pull): + # f+++++++++ newfile (new file, pull) + # f'*) changed+="$name"$'\n' ;; + *) : ;; # dir entries, metadata-only, unknown + esac + done + CHANGED_FILES_BY_KIND[$kind]="$changed" + DELETED_FILES_BY_KIND[$kind]="$deleted" + if [ -n "$changed$deleted" ]; then any_change=1; fi +done + +if [ "$any_change" -eq 0 ]; then + echo "up to date: $HOST/$STACK is already in sync with server." + # Diagnostic: if the remote dir actually looks empty, we may have been + # fooled by an rsync quirk — dump what rsync saw so the user can tell. + for entry in "${PAIRS[@]}"; do + IFS='|' read -r kind _ _ <<<"$entry" + raw=${RAW_RSYNC_OUT_BY_KIND[$kind]:-} + remote_path="/opt/docker/$kind/$STACK/" + remote_count=$(ssh -o BatchMode=yes "$TARGET" \ + "find '$remote_path' -mindepth 1 -maxdepth 1 2>/dev/null | wc -l" \ + 2>/dev/null || echo "?") + printf ' %s: remote has %s entries, rsync itemize output:\n' "$kind" "$remote_count" + if [ -z "$raw" ]; then + printf ' (empty — rsync reported no work)\n' + else + sed 's/^/ /' <<<"$raw" + fi + done + exit 0 +fi + +# --------- Print diffs. ----------------------------------------------- +divider() { printf '\n%s\n' "------------------------------------------------------------"; } + +for entry in "${PAIRS[@]}"; do + IFS='|' read -r kind src dest <<<"$entry" + remote_base="/opt/docker/$kind/$STACK" + changed=${CHANGED_FILES_BY_KIND[$kind]:-} + deleted=${DELETED_FILES_BY_KIND[$kind]:-} + [ -z "$changed$deleted" ] && continue + + printf '\n=== %s → %s ===\n' "$src" "$dest" + + while IFS= read -r rel; do + [ -z "$rel" ] && continue + local_file="$src$rel" + remote_file="$remote_base/$rel" + divider + if ssh -o BatchMode=yes "$TARGET" "[ -f '$remote_file' ]" 2>/dev/null; then + printf 'MODIFY %s\n' "$rel" + diff -u --label "a/$rel (remote)" --label "b/$rel (local)" \ + <(ssh -o BatchMode=yes "$TARGET" "cat '$remote_file'" 2>/dev/null) \ + "$local_file" || true + else + printf 'ADD %s\n' "$rel" + diff -u --label /dev/null --label "b/$rel (local)" \ + /dev/null "$local_file" || true + fi + done <<<"$changed" + + while IFS= read -r rel; do + [ -z "$rel" ] && continue + remote_file="$remote_base/$rel" + divider + printf 'DELETE %s\n' "$rel" + diff -u --label "a/$rel (remote)" --label /dev/null \ + <(ssh -o BatchMode=yes "$TARGET" "cat '$remote_file'" 2>/dev/null) \ + /dev/null || true + done <<<"$deleted" +done + +divider + +if [ "$ASSUME_YES" -ne 1 ]; then + read -r -p "Apply these changes to $TARGET? [y/N] " ans + case "$ans" in + y|Y|yes|YES) ;; + *) echo "aborted."; exit 1 ;; + esac +fi + +# --------- Apply. ----------------------------------------------------- +for entry in "${PAIRS[@]}"; do + IFS='|' read -r kind src dest <<<"$entry" + extra=() + [ "$kind" = compose ] && extra+=(--exclude='conf/') + printf 'pushing %s → %s\n' "$src" "$dest" + rsync -az --delete \ + "${EXCLUDES[@]}" "${extra[@]}" \ + "$src" "$dest" +done + +echo "done." diff --git a/scripts/refresh-server-info.sh b/scripts/refresh-server-info.sh new file mode 100755 index 0000000..909f70a --- /dev/null +++ b/scripts/refresh-server-info.sh @@ -0,0 +1,260 @@ +#!/usr/bin/env bash +# refresh-server-info.sh — pull a fresh system-details.txt from servers. +# +# Hosts are discovered by listing `servers/*/` directory names. Each name +# is used as the SSH target, so put matching entries in ~/.ssh/config to +# customize user / port / identity. +# +# Fallback: if the dir name doesn't resolve (and ssh_config doesn't rewrite +# it), the script looks for `servers//ssh-target` (one line, containing +# an IP or `user@ip`) and uses that instead. This keeps the tool working +# from a fresh clone without requiring DNS or ssh_config setup. +# +# For each host: +# 1. Run scripts/server_inspect.sh on the remote via `ssh 'bash -s'`. +# 2. Write output atomically to `servers//system-details.txt`. +# A failed SSH/run never clobbers the previous good snapshot. +# +# Exit status is non-zero if any host failed. +# +# Usage: +# scripts/refresh-server-info.sh show this help +# scripts/refresh-server-info.sh all refresh every host +# scripts/refresh-server-info.sh ana-docker refresh one host +# scripts/refresh-server-info.sh ana-docker ana-ml2 refresh several +# scripts/refresh-server-info.sh --dry-run all preview, no ssh +# scripts/refresh-server-info.sh --validate-only all checks only +# scripts/refresh-server-info.sh --validate-only checks one host +# +# Discovery is validated before any ssh is attempted. Each host prints +# an indented "! " line per warning (unreadable files, empty or +# missing ssh-target, unresolvable name with no fallback, missing README +# or system-details, etc). Warnings never block the refresh — they're +# informational — but are surfaced so drift is visible. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +INSPECT="$SCRIPT_DIR/server_inspect.sh" +SERVERS_DIR="$REPO_ROOT/servers" + +if [ ! -f "$INSPECT" ]; then + echo "error: $INSPECT not found" >&2 + exit 2 +fi + +DRY_RUN=0 +VALIDATE_ONLY=0 +REQUESTED=() +for arg in "$@"; do + case "$arg" in + --dry-run) DRY_RUN=1 ;; + --validate-only|--validate) VALIDATE_ONLY=1 ;; + -h|--help) + sed -n '2,32p' "$0" + exit 0 + ;; + -*) echo "error: unknown flag $arg" >&2; exit 2 ;; + *) REQUESTED+=("$arg") ;; + esac +done + +discover_hosts() { + find "$SERVERS_DIR" -mindepth 1 -maxdepth 1 -type d -printf '%f\n' | sort +} + +# No positional args → show help. Fleet-wide operations must be explicit. +if [ "${#REQUESTED[@]}" -eq 0 ]; then + sed -n '2,32p' "$0" + exit 0 +fi + +# The literal keyword "all" expands to every discovered host. Using it as +# a sentinel (rather than making "no args" mean "all") makes fleet-wide +# runs deliberate — useful when you might otherwise hit 20 hosts by accident. +if [ "${#REQUESTED[@]}" -eq 1 ] && [ "${REQUESTED[0]}" = "all" ]; then + mapfile -t HOSTS < <(discover_hosts) +else + # Check for 'all' mixed with other names — almost certainly a mistake. + for r in "${REQUESTED[@]}"; do + if [ "$r" = "all" ]; then + echo "error: 'all' must be the only argument when used" >&2 + exit 2 + fi + done + HOSTS=("${REQUESTED[@]}") +fi + +if [ "${#HOSTS[@]}" -eq 0 ]; then + echo "error: no hosts found under $SERVERS_DIR" >&2 + exit 2 +fi + +resolve_target() { + # Prints the SSH target to use for a given dir name. + # Preference order: + # 1. The dir name — if ssh's effective hostname (after ssh_config) resolves. + # 2. Contents of servers//ssh-target (first whitespace token). + # 3. The dir name as-is — let ssh fail with its own error. + local host="$1" + local effective + effective=$(ssh -G "$host" 2>/dev/null | awk '/^hostname /{print $2; exit}') + if [ -n "$effective" ] && getent hosts "$effective" >/dev/null 2>&1; then + echo "$host" + return + fi + local fallback="$SERVERS_DIR/$host/ssh-target" + if [ -f "$fallback" ]; then + awk 'NF{print $1; exit}' "$fallback" + return + fi + echo "$host" +} + +validate_host() { + # Prints one warning per line. Empty output = clean. + local host="$1" + local dir="$SERVERS_DIR/$host" + + if [ ! -d "$dir" ]; then + printf '%s\n' "dir missing: $dir" + return + fi + if [ ! -r "$dir" ] || [ ! -x "$dir" ]; then + printf '%s\n' "dir not readable/searchable (check permissions)" + return + fi + + local stf="$dir/ssh-target" + if [ -e "$stf" ]; then + if [ ! -f "$stf" ]; then + printf '%s\n' "ssh-target is not a regular file" + elif [ ! -r "$stf" ]; then + printf '%s\n' "ssh-target not readable" + elif [ ! -s "$stf" ]; then + printf '%s\n' "ssh-target is empty" + else + local t + t=$(awk 'NF{print $1; exit}' "$stf" 2>/dev/null || true) + if [ -z "$t" ]; then + printf '%s\n' "ssh-target has no non-blank content" + elif [[ "$t" =~ [[:space:]] ]]; then + printf '%s\n' "ssh-target first token contains whitespace ('$t')" + fi + fi + fi + + local effective="" + effective=$(ssh -G "$host" 2>/dev/null | awk '/^hostname /{print $2; exit}') + local resolves=0 + if [ -n "$effective" ] && getent hosts "$effective" >/dev/null 2>&1; then + resolves=1 + fi + if [ "$resolves" -eq 0 ] && [ ! -s "$stf" ]; then + printf '%s\n' "name '$host' does not resolve and no ssh-target fallback present" + fi + + [ -f "$dir/README.md" ] || printf '%s\n' "README.md missing" + if [ ! -f "$dir/system-details.txt" ]; then + printf '%s\n' "system-details.txt missing (never refreshed)" + else + validate_snapshot "$dir/system-details.txt" + fi +} + +validate_snapshot() { + # Read the captured system-details.txt and yield a warning for each + # known problem marker that indicates the remote inspect ran but didn't + # have the rights / environment it needed. + local file="$1" + if grep -q 'docker daemon not reachable by current user' "$file"; then + printf '%s\n' "remote user cannot reach docker daemon (add to 'docker' group or fix socket perms)" + fi + if grep -qE '^Server:[[:space:]]*$' "$file" \ + && ! grep -q 'docker daemon not reachable by current user' "$file"; then + printf '%s\n' "docker 'Server:' line is blank (daemon down or inaccessible)" + fi + if grep -q '^nvidia-smi present, but failed' "$file"; then + printf '%s\n' "nvidia-smi failed on remote (driver broken or no permission)" + fi + if ! grep -q '^===== DONE =====' "$file"; then + printf '%s\n' "snapshot appears truncated (missing trailing '===== DONE =====' marker)" + fi +} + +print_warnings() { + # $1 = indent, rest = warning lines + local indent="$1"; shift + local w + for w in "$@"; do + printf '%s! %s\n' "$indent" "$w" + done +} + +pad=0 +for h in "${HOSTS[@]}"; do (( ${#h} > pad )) && pad=${#h}; done + +if [ "$VALIDATE_ONLY" -eq 1 ]; then + printf 'Validating %d host(s):\n' "${#HOSTS[@]}" + total_warn=0 + for host in "${HOSTS[@]}"; do + mapfile -t warnings < <(validate_host "$host") + if [ "${#warnings[@]}" -eq 0 ]; then + printf ' %-*s ok\n' "$pad" "$host" + else + printf ' %-*s %d warning(s)\n' "$pad" "$host" "${#warnings[@]}" + print_warnings " " "${warnings[@]}" + total_warn=$((total_warn + ${#warnings[@]})) + fi + done + if [ "$total_warn" -gt 0 ]; then exit 1; fi + exit 0 +fi + +printf 'Refreshing %d host(s):\n' "${#HOSTS[@]}" +failed=() +for host in "${HOSTS[@]}"; do + out="$SERVERS_DIR/$host/system-details.txt" + tmp="$out.new" + + mapfile -t warnings < <(validate_host "$host") + + target=$(resolve_target "$host") + if [ "$target" = "$host" ]; then + label="$host" + else + label="$host → $target" + fi + + printf ' %-*s ' "$pad" "$label" + + if [ "$DRY_RUN" -eq 1 ]; then + printf 'would run: ssh %s bash -s < %s > %s\n' "$target" "$INSPECT" "$out" + [ "${#warnings[@]}" -gt 0 ] && print_warnings " " "${warnings[@]}" + continue + fi + + mkdir -p "$SERVERS_DIR/$host" + + if ssh -o BatchMode=yes -o ConnectTimeout=10 "$target" 'bash -s' < "$INSPECT" > "$tmp" 2> "$tmp.err"; then + mv "$tmp" "$out" + rm -f "$tmp.err" + bytes=$(wc -c < "$out") + printf 'ok (%s bytes)\n' "$bytes" + else + rc=$? + rm -f "$tmp" + err=$(head -n 1 "$tmp.err" 2>/dev/null || true) + rm -f "$tmp.err" + printf 'FAIL (rc=%d) %s\n' "$rc" "$err" + failed+=("$host") + fi + + [ "${#warnings[@]}" -gt 0 ] && print_warnings " " "${warnings[@]}" +done + +if [ "${#failed[@]}" -gt 0 ]; then + printf '\n%d host(s) failed: %s\n' "${#failed[@]}" "${failed[*]}" >&2 + exit 1 +fi diff --git a/scripts/server_inspect.sh b/scripts/server_inspect.sh new file mode 100755 index 0000000..c6a4df8 --- /dev/null +++ b/scripts/server_inspect.sh @@ -0,0 +1,166 @@ +#!/usr/bin/env bash +# server_inspect.sh — collect server details for writing Docker Compose files. +# +# Conventions this script assumes and reports on: +# - Compose files: /opt/docker/compose//{docker-compose.yml,compose.yaml} +# - Config mounts: /opt/docker/conf//... +# - Named volumes preferred over bind mounts for persistent state. +# +# Safe: read-only. No modifications are made. +# Usage: +# bash server_inspect.sh # print to stdout +# bash server_inspect.sh /tmp/report.txt # also save to file + +set -u + +OUT="${1:-}" +if [ -n "$OUT" ]; then exec > >(tee "$OUT") 2>&1; fi + +hr() { printf '\n===== %s =====\n\n' "$*"; } +sub() { printf '\n----- %s -----\n' "$*"; } +have() { command -v "$1" >/dev/null 2>&1; } + +# --- HOST -------------------------------------------------------------------- +hr "HOST" +echo "Hostname: $(hostname -f 2>/dev/null || hostname)" +echo "Date: $(date -Iseconds)" +echo "Uptime: $(uptime -p 2>/dev/null || uptime)" +if [ -f /etc/os-release ]; then + . /etc/os-release + echo "OS: ${PRETTY_NAME:-unknown}" +fi +echo "Kernel: $(uname -r)" +echo "Arch: $(uname -m)" + +# --- HARDWARE ---------------------------------------------------------------- +hr "HARDWARE" +if [ -f /proc/cpuinfo ]; then + echo "CPU cores: $(grep -c ^processor /proc/cpuinfo)" + echo "CPU model: $(awk -F: '/model name/ {print $2; exit}' /proc/cpuinfo | sed 's/^[[:space:]]*//')" +fi +if [ -f /proc/meminfo ]; then + awk '/^MemTotal:|^MemAvailable:/ {printf "%-11s %.1f GB\n", $1, $2/1024/1024}' /proc/meminfo +fi + +# --- GPUS -------------------------------------------------------------------- +hr "GPUS" +if have nvidia-smi; then + nvidia-smi --query-gpu=index,name,memory.total,memory.free,driver_version --format=csv +else + echo "nvidia-smi not present (no NVIDIA GPUs or driver not installed)" +fi + +# --- FILESYSTEM -------------------------------------------------------------- +hr "FILESYSTEMS (df)" +df -h -x tmpfs -x devtmpfs -x overlay 2>/dev/null + +hr "PERSISTENT MOUNTS (/etc/fstab, non-comment)" +if [ -r /etc/fstab ]; then + grep -vE '^\s*(#|$)' /etc/fstab +fi + +hr "TARGETED DATA PATHS" +for path in /tank /models /opt /opt/docker /opt/docker/compose /opt/docker/conf \ + /var/lib/docker /data /srv; do + if [ -d "$path" ]; then + size=$(du -sh "$path" 2>/dev/null | awk '{print $1}') + echo "$path (total: ${size:-?})" + ls -la --time-style=long-iso "$path" 2>/dev/null | sed 's/^/ /' | head -30 + echo + fi +done + +# --- DOCKER ------------------------------------------------------------------ +hr "DOCKER" +if ! have docker; then + echo "docker not installed" +else + docker version --format 'Server: {{.Server.Version}} Client: {{.Client.Version}}' 2>/dev/null \ + || echo "docker daemon not reachable by current user" + + sub "docker info" + docker info --format 'Containers: {{.Containers}} (running {{.ContainersRunning}}, paused {{.ContainersPaused}}, stopped {{.ContainersStopped}}) +Images: {{.Images}} +Runtimes: {{.Runtimes}} +Default runtime: {{.DefaultRuntime}} +Storage driver: {{.Driver}} +Root dir: {{.DockerRootDir}} +Server version: {{.ServerVersion}}' 2>/dev/null + + sub "running containers" + docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}' 2>/dev/null + + sub "all containers" + docker ps -a --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}' 2>/dev/null + + sub "networks" + docker network ls --format 'table {{.Name}}\t{{.Driver}}\t{{.Scope}}' 2>/dev/null + + sub "networks (external, non-default — worth knowing for compose external: true)" + docker network ls --filter driver=bridge --format '{{.Name}}' 2>/dev/null \ + | grep -vE '^(bridge|host|none)$' || true + + sub "named volumes" + docker volume ls --format 'table {{.Name}}\t{{.Driver}}' 2>/dev/null + + sub "compose projects currently running" + docker ps --format '{{.Label "com.docker.compose.project"}}' 2>/dev/null \ + | sort -u | grep -v '^$' || echo "(none)" +fi + +# --- COMPOSE FILES ----------------------------------------------------------- +hr "COMPOSE FILES (/opt/docker/compose/)" +if [ -d /opt/docker/compose ]; then + find /opt/docker/compose -maxdepth 3 -type f \ + \( -name 'docker-compose.y*ml' -o -name 'compose.y*ml' \) 2>/dev/null \ + | sort | while read -r f; do + printf '\n>>> %s\n' "$f" + cat "$f" + done +else + echo "/opt/docker/compose not present" +fi + +# --- CONFIG LAYOUT ----------------------------------------------------------- +hr "CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries)" +if [ -d /opt/docker/conf ]; then + find /opt/docker/conf -maxdepth 4 2>/dev/null | sort | head -200 +else + echo "/opt/docker/conf not present" +fi + +# --- PORTS ------------------------------------------------------------------- +hr "LISTENING PORTS" +if have ss; then + ss -tlnH 2>/dev/null | awk '{print $4}' | sort -u +elif have netstat; then + netstat -tln 2>/dev/null | awk 'NR>2 {print $4}' | sort -u +else + echo "ss and netstat both unavailable" +fi + +# --- HF / MODEL CACHES ------------------------------------------------------- +hr "MODEL / HUGGINGFACE CACHES" +for path in /tank/aimodels/huggingface /tank/aimodels/llm ~/.cache/huggingface \ + /data/huggingface /opt/huggingface; do + if [ -d "$path" ]; then + size=$(du -sh "$path" 2>/dev/null | awk '{print $1}') + echo "$path (${size:-?})" + if [ -d "$path/hub" ]; then + echo " hub entries:" + ls "$path/hub" 2>/dev/null | sed 's/^/ /' | head -30 + fi + echo + fi +done + +# --- SYSTEMD SERVICES (docker-adjacent) -------------------------------------- +hr "DOCKER-ADJACENT SYSTEMD SERVICES" +if have systemctl; then + systemctl list-units --type=service --state=running --no-pager --no-legend 2>/dev/null \ + | awk '{print $1, $4}' \ + | grep -Ei 'docker|container|traefik|nvidia' || echo "(none matching)" +fi + +hr "DONE" +echo "Paste the above back into the chat, or pass a path as argv[1] to save." diff --git a/scripts/sync-stacks.sh b/scripts/sync-stacks.sh new file mode 100755 index 0000000..11dcc8e --- /dev/null +++ b/scripts/sync-stacks.sh @@ -0,0 +1,190 @@ +#!/usr/bin/env bash +# sync-stacks.sh — pull /opt/docker/{compose,conf}// from every +# server into version-controlled `stacks-mirror///`. +# +# Layout (flat per stack): +# stacks-mirror/// <- mirrors /opt/docker/compose// +# stacks-mirror///conf/ <- mirrors /opt/docker/conf// +# +# Opt-out (per-stack, per-kind): +# stacks-mirror///.no-sync → skip stack entirely +# stacks-mirror///conf/.no-sync → skip conf only +# The marker file is preserved; only the rsync is suppressed. Create the +# marker manually for any stack you don't want mirrored. +# +# Secrets and runtime state are always excluded regardless of opt-out: +# .env, .env.*, acme.json, client_secrets.json, +# *.pem, *.key, *.crt, *.pfx, +# *.sqlite, *.sqlite3, *.db, *.log, *.log.*, *.pid, +# hub/, logs/ +# +# Usage: +# scripts/sync-stacks.sh # pull from every discovered host +# scripts/sync-stacks.sh ana-docker nh3-docker +# scripts/sync-stacks.sh --dry-run # show what would change, no writes + +set -euo pipefail + +if ! command -v rsync >/dev/null 2>&1; then + echo "error: rsync is not installed on this workstation" >&2 + echo " install it (e.g. 'sudo apt install rsync') and ensure every remote host has it too" >&2 + exit 2 +fi + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +SERVERS_DIR="$REPO_ROOT/servers" +MIRROR_DIR="$REPO_ROOT/stacks-mirror" + +EXCLUDES=( + # Include .env.example / *.env.example templates before the broader + # .env* exclude — rsync processes these in order, first match wins. + --include='.env.example' + --include='*.env.example' + --exclude=.env + --exclude='.env.*' + --exclude=acme.json + --exclude=client_secrets.json + --exclude='*.pem' + --exclude='*.key' + --exclude='*.crt' + --exclude='*.pfx' + --exclude='*.sqlite' + --exclude='*.sqlite3' + --exclude='*.db' + --exclude='*.log' + --exclude='*.log.*' + --exclude='*.pid' + --exclude='hub/' + --exclude='logs/' +) + +DRY_RUN=0 +REQUESTED=() +for arg in "$@"; do + case "$arg" in + --dry-run) DRY_RUN=1 ;; + -h|--help) sed -n '2,24p' "$0"; exit 0 ;; + -*) echo "error: unknown flag $arg" >&2; exit 2 ;; + *) REQUESTED+=("$arg") ;; + esac +done + +resolve_target() { + local host="$1" + local effective + effective=$(ssh -G "$host" 2>/dev/null | awk '/^hostname /{print $2; exit}') + if [ -n "$effective" ] && getent hosts "$effective" >/dev/null 2>&1; then + echo "$host"; return + fi + local fb="$SERVERS_DIR/$host/ssh-target" + if [ -f "$fb" ]; then awk 'NF{print $1; exit}' "$fb"; return; fi + echo "$host" +} + +list_remote_subdirs() { + # $1 = ssh target, $2 = remote parent path + ssh -o BatchMode=yes -o ConnectTimeout=10 "$1" \ + "find '$2' -maxdepth 1 -mindepth 1 -type d -printf '%f\n' 2>/dev/null | sort" \ + 2>/dev/null || true +} + +sync_one() { + # $1 = host, $2 = ssh target, $3 = stack, + # $4 = 'compose'|'conf' (kind), + # $5 = local dest dir + local host="$1" target="$2" stack="$3" kind="$4" dest="$5" + local remote_src="/opt/docker/$kind/$stack/" + local skip="$dest/.no-sync" + local label + if [ "$kind" = conf ]; then label='conf '; else label='compose'; fi + + mkdir -p "$dest" + + if [ -f "$skip" ]; then + printf ' %s skip (.no-sync)\n' "$label" + return 0 + fi + + local extra=() + # Don't recurse into conf/ from the compose side — it's its own mirror target. + [ "$kind" = compose ] && extra+=(--exclude='conf/') + [ "$DRY_RUN" -eq 1 ] && extra+=(--dry-run) + + local err rc=0 + err=$( + rsync -az --delete --info=stats0,flist0 \ + "${EXCLUDES[@]}" "${extra[@]}" \ + "$target:$remote_src" "$dest/" 2>&1 + ) || rc=$? + + if [ $rc -ne 0 ]; then + printf ' %s FAIL (rc=%d) %s\n' "$label" "$rc" "$(echo "$err" | head -n 1)" + return 1 + fi + if [ "$DRY_RUN" -eq 1 ]; then + printf ' %s dry-run ok\n' "$label" + else + printf ' %s ok\n' "$label" + fi + return 0 +} + +if [ "${#REQUESTED[@]}" -eq 0 ]; then + mapfile -t HOSTS < <(find "$SERVERS_DIR" -mindepth 1 -maxdepth 1 -type d -printf '%f\n' | sort) +else + HOSTS=("${REQUESTED[@]}") +fi + +[ "${#HOSTS[@]}" -eq 0 ] && { echo "error: no hosts found" >&2; exit 2; } +[ "$DRY_RUN" -eq 1 ] && echo "(dry-run)" + +mkdir -p "$MIRROR_DIR" +total_fail=0 + +for host in "${HOSTS[@]}"; do + target=$(resolve_target "$host") + printf '%s (%s):\n' "$host" "$target" + + mapfile -t compose_stacks < <(list_remote_subdirs "$target" /opt/docker/compose) + mapfile -t conf_stacks < <(list_remote_subdirs "$target" /opt/docker/conf) + + if [ "${#compose_stacks[@]}" -eq 0 ] && [ "${#conf_stacks[@]}" -eq 0 ]; then + printf ' (no stacks discovered — check ssh + remote /opt/docker layout)\n' + continue + fi + + # Union of stack names. + mapfile -t all_stacks < <(printf '%s\n' "${compose_stacks[@]}" "${conf_stacks[@]}" | sort -u | grep .) + + # Warn about local stacks that no longer exist on the remote. + if [ -d "$MIRROR_DIR/$host" ]; then + for local_stack in "$MIRROR_DIR/$host"/*/; do + [ -d "$local_stack" ] || continue + name=$(basename "$local_stack") + if ! printf '%s\n' "${all_stacks[@]}" | grep -qxF "$name"; then + printf ' ! %s exists locally but not on remote (stale — remove manually if intentional)\n' "$name" + fi + done + fi + + for stack in "${all_stacks[@]}"; do + printf ' %s\n' "$stack" + stack_root="$MIRROR_DIR/$host/$stack" + if [ -f "$stack_root/.no-sync" ]; then + printf ' skip (.no-sync at stack root)\n' + continue + fi + if printf '%s\n' "${compose_stacks[@]}" | grep -qxF "$stack"; then + sync_one "$host" "$target" "$stack" compose "$stack_root" || total_fail=$((total_fail+1)) + fi + if printf '%s\n' "${conf_stacks[@]}" | grep -qxF "$stack"; then + sync_one "$host" "$target" "$stack" conf "$stack_root/conf" || total_fail=$((total_fail+1)) + fi + done +done + +if [ "$total_fail" -gt 0 ]; then + printf '\n%d sync operation(s) failed\n' "$total_fail" >&2 + exit 1 +fi diff --git a/servers/ana-docker/README.md b/servers/ana-docker/README.md new file mode 100644 index 0000000..090075b --- /dev/null +++ b/servers/ana-docker/README.md @@ -0,0 +1,67 @@ +# ana-docker + +General-purpose Docker host for the Anaheim colo. Runs everything at `10.250.0.0/16` that doesn't need a GPU — pair it with ana-ml2, which handles the GPU workloads. + +## Network + +- **LAN IP:** 10.250.50.70 +- **FQDN:** `ana-docker.phasefinal.com` +- **SSH:** standard port 22 +- **Traefik entrypoints:** 80/443 terminate here; cert resolver `anaprod` (Let's Encrypt) + +## Hardware + +- **CPU:** 8 vCPU (QEMU virtual — this is a VM) +- **RAM:** 15.6 GB +- **GPUs:** none +- **Storage:** 245 GB root (ext4) + NFS mounts from `10.250.50.50` (TrueNAS) +- **OS:** Debian 12 (bookworm), kernel 6.1.x +- **Docker:** 20.10.24 + +## Key paths + +| Path | Purpose | +|------|---------| +| `/opt/docker/compose//` | Compose files (also a git repo) | +| `/opt/docker/conf//` | Config bind mounts | +| `/opt/docker/data/` | Service state (legacy — most stacks now use named volumes) | +| `/mnt/compose` | NFS — `10.250.50.50:/mnt/docker` | +| `/mnt/backup` | NFS — `10.250.50.50:/mnt/backup` (restic target) | +| `/mnt/tnvms` | NFS — `10.250.50.50:/mnt/pve-VMStorage` | + +## Running stacks + +| Stack | Port(s) | Notes | +|-------|---------|-------| +| traefik | 80 / 443 / 8380 | Reverse proxy + TLS (cert resolver `anaprod`) | +| crowdsec (+ blocklist-mirror) | 41412 | IPS; bouncer runs as a Traefik plugin | +| gitea | 3000 / 222 | Git hosting — `gitea.phasefinal.com` | +| vaultwarden | 9080 | Password vault — `vaultwarden.phasefinal.com` | +| synapse (+ synapse-db, element-web) | internal | Matrix homeserver — `matrix.phasefinal.com`, client at `chat.phasefinal.com` | +| seafile (+ mariadb, memcached) | 9180 | File sync — `seafile.phasefinal.com` | +| searxng | 9996 | Private search metaengine | +| openwebui | 3100 | Chat UI frontend | +| sillytavern | 8100 | Chat UI | +| mailrise | 8025 | SMTP-to-notification gateway | +| rustdesk (hbbs + hbbr) | host-net 21115-21119 | Remote desktop relay — `rustdesk.phasefinal.com` | +| dockge | 5001 | Docker stack management UI | +| beszel | 8090 | Fleet metrics hub (+ local agent); agents on the other hosts report here | +| dozzle (hub as `dozzle-hub`) | 8088 | Fleet log viewer; agents on the other hosts report here | +| restic rest-server | 8000 | Anaheim-side restic endpoint (writes to TrueNAS NFS at `/mnt/backup/restic/repo/ana/`); paired with `rest-server-nh3` on the Synology for the NH3 side | +| backrest | 9898 | Fleet-wide restic snapshot viewer / restore UI — points at both rest-servers | +| it-tools | 8780 | Dev utilities | +| mattermost | — | Stopped; kept around for reference | + +Portainer was retired from this host; stack management is now handled via Dockge + Beszel. + +## Refresh state + +```bash +scripts/refresh-server-info.sh ana-docker +``` + +Latest snapshot: `system-details.txt` (regenerate as needed). + +## Placement rule + +If a new stack needs a GPU it goes on **ana-ml2**, otherwise it lands here. diff --git a/servers/ana-docker/ssh-target b/servers/ana-docker/ssh-target new file mode 100644 index 0000000..f24499d --- /dev/null +++ b/servers/ana-docker/ssh-target @@ -0,0 +1 @@ +10.250.50.70 diff --git a/servers/ana-docker/system-details.txt b/servers/ana-docker/system-details.txt new file mode 100644 index 0000000..c1a7777 --- /dev/null +++ b/servers/ana-docker/system-details.txt @@ -0,0 +1,1317 @@ + +===== HOST ===== + +Hostname: ana-docker.phasefinal.com +Date: 2026-04-19T22:15:55-07:00 +Uptime: up 2 weeks, 5 days, 23 hours, 24 minutes +OS: Debian GNU/Linux 12 (bookworm) +Kernel: 6.1.0-44-amd64 +Arch: x86_64 + +===== HARDWARE ===== + +CPU cores: 8 +CPU model: QEMU Virtual CPU version 2.5+ +MemTotal: 15.6 GB +MemAvailable: 10.5 GB + +===== GPUS ===== + +nvidia-smi not present (no NVIDIA GPUs or driver not installed) + +===== FILESYSTEMS (df) ===== + +Filesystem Size Used Avail Use% Mounted on +/dev/sda1 245G 79G 154G 34% / +10.250.50.50:/mnt/docker 20T 0 20T 0% /mnt/compose +10.250.50.50:/mnt/backup 20T 1.1G 20T 1% /mnt/backup +10.250.50.50:/mnt/pve-VMStorage 22T 2.4T 20T 11% /mnt/tnvms + +===== PERSISTENT MOUNTS (/etc/fstab, non-comment) ===== + +UUID=1f0fecb0-efcf-4403-a180-f67cb4615f48 / ext4 errors=remount-ro 0 1 +UUID=ab0fa49a-61d4-4b2a-af7e-bc32f84bbd3b none swap sw 0 0 +/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0 +10.250.50.50:/mnt/docker /mnt/tndocker nfs noauto,x-systemd.automount 0 0 +10.250.50.50:/mnt/pve-VMStorage /mnt/tnvms nfs noauto,x-systemd.automount 0 0 +10.250.50.50:/mnt/docker /mnt/compose nfs defaults 0 0 +10.250.50.50:/mnt/backup /mnt/backup nfs defaults 0 0 + +===== TARGETED DATA PATHS ===== + +/opt (total: 6.1G) + total 16 + drwxr-xr-x 4 root root 4096 2026-04-02 22:12 . + drwxr-xr-x 20 root root 4096 2026-03-30 22:49 .. + drwxr-xr-x 10 lkraven root 4096 2026-04-03 08:44 AIPA + drwxrwxrwx 6 root root 4096 2026-04-15 23:28 docker + +/opt/docker (total: 6.0G) + total 32 + drwxrwxrwx 6 root root 4096 2026-04-15 23:28 . + drwxr-xr-x 4 root root 4096 2026-04-02 22:12 .. + drwxr-xr-x 20 lkraven lkraven 4096 2026-04-19 19:41 compose + drwxr-xr-x 10 lkraven lkraven 4096 2026-04-18 18:57 conf + drwxr-xr-x 7 lkraven lkraven 4096 2025-02-05 19:28 data + drwxr-xr-x 8 lkraven lkraven 4096 2026-04-15 23:47 .git + -rw-r--r-- 1 lkraven lkraven 14 2024-05-30 18:44 .gitignore + -rw-r--r-- 1 lkraven lkraven 60 2024-05-30 15:33 README.md + +/opt/docker/compose (total: 224K) + total 84 + drwxr-xr-x 20 lkraven lkraven 4096 2026-04-19 19:41 . + drwxrwxrwx 6 root root 4096 2026-04-15 23:28 .. + drwxr-xr-x 2 lkraven lkraven 4096 2026-04-19 22:11 backrest + drwxr-xr-x 2 root root 4096 2026-04-19 00:50 beszel + drwxr-xr-x 2 root root 4096 2026-04-15 23:02 crowdsec + drwxr-xr-x 2 lkraven lkraven 4096 2024-05-30 18:50 dockge + drwxr-xr-x 2 root root 4096 2026-04-19 00:25 dozzle-hub + drwxr-xr-x 2 lkraven lkraven 4096 2024-05-30 18:50 gitea + drwxr-xr-x 2 root root 4096 2025-03-18 22:07 ittools + drwxr-xr-x 2 lkraven lkraven 4096 2024-05-30 18:50 mailrise + drwxr-xr-x 2 lkraven lkraven 4096 2024-05-30 18:50 mattermost + drwxr-xr-x 2 root root 4096 2025-02-05 21:53 openwebui + -rw-r--r-- 1 lkraven lkraven 25 2024-05-30 15:33 README.md + drwxr-xr-x 2 lkraven lkraven 4096 2024-05-30 18:50 restic + drwxr-xr-x 2 lkraven lkraven 4096 2024-05-30 18:50 rustdesk + drwxr-xr-x 2 lkraven lkraven 4096 2024-05-30 18:50 seafile + drwxr-xr-x 2 root root 4096 2026-04-18 18:55 searxng + drwxr-xr-x 2 root root 4096 2025-02-05 19:29 sillytavern + drwxr-xr-x 2 linus linus 4096 2026-04-11 13:59 synapse + drwxr-xr-x 2 lkraven lkraven 4096 2024-05-30 18:50 traefik + drwxr-xr-x 2 lkraven lkraven 4096 2024-05-30 18:50 vaultwarden + +/opt/docker/conf (total: 3.4M) + total 40 + drwxr-xr-x 10 lkraven lkraven 4096 2026-04-18 18:57 . + drwxrwxrwx 6 root root 4096 2026-04-15 23:28 .. + drwxr-xr-x 11 lkraven root 4096 2026-04-16 17:56 crowdsec + drwxr-xr-x 2 lkraven lkraven 4096 2024-05-30 16:16 mailrise + drwxr-xr-x 2 lkraven lkraven 4096 2024-05-30 16:19 mattermost + drwxr-xr-x 2 lkraven lkraven 4096 2026-04-18 19:22 searxng + drwxr-xr-x 2 lkraven lkraven 4096 2025-03-18 22:50 sillytavern + drwxr-xr-x 2 linus linus 4096 2026-04-15 21:48 synapse + drwxr-xr-x 3 lkraven lkraven 4096 2024-05-30 16:11 traefik-ana + drwxr-xr-x 2 lkraven lkraven 4096 2026-04-16 16:28 vaultwarden + +/var/lib/docker (total: 4.0K) + +/data (total: 60K) + total 12 + drwxr-xr-x 3 root root 4096 2023-06-12 22:33 . + drwxr-xr-x 20 root root 4096 2026-03-30 22:49 .. + drwxr-xr-x 4 root root 4096 2023-11-02 17:05 compose + +/srv (total: 4.0K) + total 8 + drwxr-xr-x 2 root root 4096 2023-04-03 00:06 . + drwxr-xr-x 20 root root 4096 2026-03-30 22:49 .. + + +===== DOCKER ===== + +Server: 20.10.24+dfsg1 Client: 20.10.24+dfsg1 + +----- docker info ----- +Containers: 26 (running 24, paused 0, stopped 2) +Images: 62 +Runtimes: map[io.containerd.runc.v2:{runc [] } io.containerd.runtime.v1.linux:{runc [] } runc:{runc [] }] +Default runtime: runc +Storage driver: overlay2 +Root dir: /var/lib/docker +Server version: 20.10.24+dfsg1 + +----- running containers ----- +NAMES IMAGE STATUS PORTS +backrest garethgeorge/backrest:latest Up 3 minutes (healthy) 0.0.0.0:9898->9898/tcp, :::9898->9898/tcp +dockge-dockge-1 louislam/dockge:latest Up 2 hours (healthy) 0.0.0.0:5001->5001/tcp, :::5001->5001/tcp +searxng searxng/searxng:latest Up 10 hours (healthy) 0.0.0.0:9996->8080/tcp, :::9996->8080/tcp +beszel henrygd/beszel:latest Up 10 hours (healthy) 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp +beszel-agent henrygd/beszel-agent:latest Up 10 hours +dozzle amir20/dozzle:latest Up 22 hours (healthy) 0.0.0.0:8088->8080/tcp, :::8088->8080/tcp +blocklist-mirror crowdsecurity/blocklist-mirror:latest Up 3 days 0.0.0.0:41412->41412/tcp, :::41412->41412/tcp +vaultwarden vaultwarden/server:latest Up 3 days (healthy) 0.0.0.0:9080->80/tcp, :::9080->80/tcp +crowdsec crowdsecurity/crowdsec:latest Up 3 days +traefik traefik:latest Up 3 days 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:8380->8080/tcp, :::8380->8080/tcp +gitea gitea/gitea:latest Up 3 days 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp, 0.0.0.0:222->22/tcp, :::222->22/tcp +element-web vectorim/element-web:v1.11.80 Up 4 days 80/tcp +synapse matrixdotorg/synapse:v1.120.0 Up 4 days (healthy) 8008-8009/tcp, 8448/tcp +synapse-db postgres:16-alpine Up 4 days (healthy) 5432/tcp +sillytavern ghcr.io/sillytavern/sillytavern:latest Up 2 weeks 0.0.0.0:8100->8000/tcp, :::8100->8000/tcp +seafile seafileltd/seafile-mc:11.0-latest Up 2 weeks 0.0.0.0:9180->80/tcp, :::9180->80/tcp +seafile-mysql mariadb:10.6 Up 2 weeks 3306/tcp +openwebui-open-webui-1 ghcr.io/open-webui/open-webui:main Up 2 weeks (healthy) 0.0.0.0:3100->8080/tcp, :::3100->8080/tcp +rest_server restic/rest-server Up 2 weeks 0.0.0.0:8000->8000/tcp, :::8000->8000/tcp +hbbs rustdesk/rustdesk-server:latest Up 2 weeks +hbbr rustdesk/rustdesk-server:latest Up 2 weeks +it-tools corentinth/it-tools:latest Up 2 weeks 0.0.0.0:8780->80/tcp, :::8780->80/tcp +seafile-memcached memcached:1.6.18 Up 2 weeks 11211/tcp +mailrise yoryan/mailrise:latest Up 2 weeks 0.0.0.0:8025->8025/tcp, :::8025->8025/tcp + +----- all containers ----- +NAMES IMAGE STATUS +backrest garethgeorge/backrest:latest Up 3 minutes (healthy) +dockge-dockge-1 louislam/dockge:latest Up 2 hours (healthy) +searxng searxng/searxng:latest Up 10 hours (healthy) +beszel henrygd/beszel:latest Up 10 hours (healthy) +beszel-agent henrygd/beszel-agent:latest Up 10 hours +dozzle amir20/dozzle:latest Up 22 hours (healthy) +blocklist-mirror crowdsecurity/blocklist-mirror:latest Up 3 days +vaultwarden vaultwarden/server:latest Up 3 days (healthy) +crowdsec crowdsecurity/crowdsec:latest Up 3 days +traefik traefik:latest Up 3 days +gitea gitea/gitea:latest Up 3 days +simple-service-bar traefik/whoami Created +simple-service-foo traefik/whoami Created +element-web vectorim/element-web:v1.11.80 Up 4 days +synapse matrixdotorg/synapse:v1.120.0 Up 4 days (healthy) +synapse-db postgres:16-alpine Up 4 days (healthy) +sillytavern ghcr.io/sillytavern/sillytavern:latest Up 2 weeks +seafile seafileltd/seafile-mc:11.0-latest Up 2 weeks +seafile-mysql mariadb:10.6 Up 2 weeks +openwebui-open-webui-1 ghcr.io/open-webui/open-webui:main Up 2 weeks (healthy) +rest_server restic/rest-server Up 2 weeks +hbbs rustdesk/rustdesk-server:latest Up 2 weeks +hbbr rustdesk/rustdesk-server:latest Up 2 weeks +it-tools corentinth/it-tools:latest Up 2 weeks +seafile-memcached memcached:1.6.18 Up 2 weeks +mailrise yoryan/mailrise:latest Up 2 weeks + +----- networks ----- +NAME DRIVER SCOPE +bridge bridge local +host host local +it-tools_default bridge local +ittools_default bridge local +mailrise_default bridge local +none null local +portainer_default bridge local +synapse_synapse-internal bridge local +traefik-net bridge local + +----- networks (external, non-default — worth knowing for compose external: true) ----- +it-tools_default +ittools_default +mailrise_default +portainer_default +synapse_synapse-internal +traefik-net + +----- named volumes ----- +VOLUME NAME DRIVER +0d2873a4373bfab9214f37bc8ad46d0ed5a90805fffa8dbef7b915493da66a7b local +1a587fdaf8f4f022cef113ce59f0ce171fcf2247c52f28df1f4e461985ff6a4f local +5b532eb6c5bb5a718a68824a004c9eee9ffb6d726a19bdf066ecb6348a6c697c local +6b7b248964ce54c151cf57a1f44cfc095d9c2bd1a9844eba2e05ad4e143e05ad local +9d1360a12bd6ca0b11d52b02a00248fd80712667510453b4ca882656798c69c8 local +28ae9933a241df783c23ff0de09882078851bef60d3440dad9243b5ba8766985 local +53bf6d927badb5904b71d4180d1f05ccac2a195b7362206a8b0ffece5771aa9a local +065fbcc9d093988c8a8c005688f119d87855407957745d82dee4ff0c4c615ea6 local +76ab976f6fb5f8f5cd322d32975206ae490f6f4644bfa315632e6ef6c3da54eb local +88b9750e458794db88b0694b5b9dd5bbe595651355f3e552bd84cbbe856fbc95 local +207f0f98a462e2b0a2da5176be52704c3aba26039f47c4ccc685619e81bf7a7a local +backrest_backrest_cache local +backrest_backrest_config local +backrest_backrest_data local +backrest_backrest_tmp local +beszel_beszel_agent_data local +beszel_beszel_data local +crowdsec_crowdsec-data local +crowdsec_traefik-logs local +dockge_dockge_data local +dozzle-hub_dozzle_data local +edf9fc3909e2faa49c147b4029db703466116f06d9fe2dc87d902bc27c6b77a0 local +gitea_gitea_data local +komodo_repo-cache local +mattermost_mattermost_data local +openwebui_open-webui local +rustdesk_rustdesk_data local +seafile_seafile_datastore local +seafile_seafile_db local +sillytavern_sillytavern_data local +sillytavern_sillytavern_extensions local +sillytavern_sillytavern_plugins local +synapse-data local +synapse_synapse-data local +synapse_synapse-db-data local +traefik_traefik-logs local +vaultwarden_vaultwarden_data local + +----- compose projects currently running ----- +backrest +beszel +crowdsec +dockge +dozzle-hub +gitea +ittools +mailrise +openwebui +restic +rustdesk +seafile +searxng +sillytavern +synapse +traefik +vaultwarden + +===== COMPOSE FILES (/opt/docker/compose/) ===== + + +>>> /opt/docker/compose/backrest/compose.yaml +# Backrest — web UI over restic repositories. +# +# Role here: single central viewer for every host's restic repo on both +# site-local S3 endpoints (TrueNAS at ana, Synology at nh3). Per-host +# `restic` runs will still be driven by systemd timers on each host; this +# stack is how we see what ran, browse snapshots, and restore. +# +# Repos and S3 credentials are configured in the Backrest UI after first +# boot — nothing baked into this file. Data (its own SQLite + queue) lives +# in a named volume so the config survives container recreation. +# +# All tunables live in .env — edit that, not this file. + +services: + backrest: + image: garethgeorge/backrest:${BACKREST_VERSION} + container_name: backrest + hostname: backrest + restart: unless-stopped + ports: + - ${BACKREST_PORT}:9898 + volumes: + - backrest_data:/data + - backrest_config:/config + - backrest_cache:/cache + - backrest_tmp:/tmp + environment: + - BACKREST_DATA=/data + - BACKREST_CONFIG=/config/config.json + - XDG_CACHE_HOME=/cache + - TMPDIR=/tmp + - TZ=${TZ:-America/Los_Angeles} + - BACKREST_PORT=0.0.0.0:9898 + healthcheck: + test: + - CMD + - wget + - -qO- + - http://localhost:9898/ + interval: 30s + timeout: 10s + retries: 3 + start_period: 15s + networks: + - tnet + labels: + - homepage.group=PFI-ANA + - homepage.name=Backrest + - homepage.icon=mdi-backup-restore + - homepage.description=Restic snapshot viewer / restore UI + - homepage.href=http://10.250.50.70:${BACKREST_PORT} +volumes: + backrest_data: null + backrest_config: null + backrest_cache: null + backrest_tmp: null +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/beszel/compose.yaml +# Beszel — lightweight server/container monitoring. +# +# Hub: single web UI with the SQLite store. Agents: per-host metric collectors +# that the hub pulls from over SSH. +# +# Multi-host layout via compose profiles: +# COMPOSE_PROFILES=hub → hub only (ana-docker) +# COMPOSE_PROFILES=hub,agent → hub + local agent on the same host +# COMPOSE_PROFILES=agent → agent only (ana-ml2) +# +# The agent uses network_mode: host so it sees real host CPU/mem/net/disk +# counters rather than container-scoped ones — that's why it can't share +# the tnet network with the hub. +# +# All tunables live in .env — edit that, not this file. + +services: + beszel: + image: henrygd/beszel:${BESZEL_VERSION} + container_name: beszel + profiles: + - hub + restart: unless-stopped + ports: + - ${BESZEL_PORT}:8090 + volumes: + - beszel_data:/beszel_data + networks: + - tnet + labels: + - homepage.group=PFI-ANA + - homepage.name=Beszel + - homepage.icon=mdi-chart-line + - homepage.description=Server + container monitoring + - homepage.href=http://10.250.50.70:${BESZEL_PORT} + healthcheck: + # The URL is relative to the container, not the host + test: ['CMD', '/beszel', 'health', '--url', 'http://localhost:8090'] + start_period: 5s # Check 5 seconds after the container starts + interval: 120s # Then check every 120 seconds after that + + beszel-agent: + image: henrygd/beszel-agent:${BESZEL_VERSION} + container_name: beszel-agent + profiles: + - agent + restart: unless-stopped + network_mode: host + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - beszel_agent_data:/var/lib/beszel-agent + environment: + - PORT=${BESZEL_AGENT_PORT:-45876} + - KEY=${BESZEL_HUB_KEY} + - HUB_URL=${HUB_URL} + - TOKEN=${BESZEL_TOKEN} + - EXTRA_FILESYSTEMS=${BESZEL_EXTRA_FS:-} +volumes: + beszel_data: null + beszel_agent_data: null +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/crowdsec/compose.yaml +# /opt/docker/compose/crowdsec/compose.yaml +# +# CrowdSec — collaborative intrusion prevention +# +# Agent: reads Docker container logs, parses events, makes decisions +# Bouncing: handled by CrowdSec Traefik plugin inside the Traefik container +# (no separate bouncer container needed) +# +# Protected services (via Traefik middleware): + +services: + crowdsec: + image: crowdsecurity/crowdsec:latest + container_name: crowdsec + restart: unless-stopped + security_opt: + - no-new-privileges:true + volumes: + - crowdsec-data:/var/lib/crowdsec/data + - /var/run/docker.sock:/var/run/docker.sock:ro + - /opt/docker/conf/crowdsec:/etc/crowdsec + environment: + - COLLECTIONS=crowdsecurity/linux crowdsecurity/traefik LePresidente/gitea + - TZ=${TZ:-America/Los_Angeles} + - CROWDSEC_LAPI_BIND=0.0.0.0:8080 + - DOCKER_API_VERSION=1.41 + networks: + - traefik-net + blocklist-mirror: + image: crowdsecurity/blocklist-mirror:latest + container_name: blocklist-mirror + restart: unless-stopped + ports: + - 41412:41412 + volumes: + - /opt/docker/conf/crowdsec/blocklist-mirror.yaml:/etc/crowdsec/bouncers/crowdsec-blocklist-mirror.yaml:ro + networks: + - traefik-net +volumes: + crowdsec-data: null +networks: + traefik-net: + external: true + +>>> /opt/docker/compose/dockge/compose.yaml +services: + dockge: + image: louislam/dockge:latest + restart: unless-stopped + ports: + # Host Port : Container Port + - 5001:5001 + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - dockge_data:/app/data + - /opt/docker/compose:/opt/docker/compose + labels: + - homepage.group=PFI-ANA + - homepage.name=Dockge + - homepage.icon=si-portainer + - homepage.description=Docker + - homepage.href=http://10.250.50.70:5001 + environment: + # Tell Dockge where is your stacks directory + - DOCKGE_STACKS_DIR=/opt/docker/compose + networks: + - tnet + +volumes: + dockge_data: null +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/dozzle-hub/compose.yaml +# Dozzle — container log viewer. +# +# Multi-host layout via compose profiles: +# COMPOSE_PROFILES=hub → runs the web UI (deploy on ana-docker) +# COMPOSE_PROFILES=agent → runs the remote agent (deploy on ana-ml2) +# +# Same compose.yaml on both servers; per-host `.env` picks the profile. +# +# All tunables live in .env — edit that, not this file. + +services: + dozzle: + image: amir20/dozzle:${DOZZLE_VERSION} + container_name: dozzle + profiles: + - hub + restart: unless-stopped + ports: + - ${DOZZLE_PORT}:8080 + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - dozzle_data:/data + environment: + - DOZZLE_HOSTNAME=${DOZZLE_HOSTNAME} + - DOZZLE_REMOTE_AGENT=${DOZZLE_REMOTE_AGENT:-} + - DOZZLE_AUTH_PROVIDER=${DOZZLE_AUTH_PROVIDER:-none} + - DOZZLE_USERNAME=${DOZZLE_USERNAME:-} + - DOZZLE_PASSWORD=${DOZZLE_PASSWORD:-} + healthcheck: + test: + - CMD + - /dozzle + - healthcheck + interval: 30s + timeout: 10s + retries: 3 + start_period: 15s + networks: + - tnet + labels: + - homepage.group=PFI-ANA + - homepage.name=Dozzle + - homepage.icon=mdi-text-box-search + - homepage.description=Container logs (ana-docker + ana-ml2) + - homepage.href=http://10.250.50.70:${DOZZLE_PORT} + dozzle-agent: + image: amir20/dozzle:${DOZZLE_VERSION} + container_name: dozzle-agent + profiles: + - agent + restart: unless-stopped + command: agent + ports: + - ${DOZZLE_AGENT_BIND:-0.0.0.0}:${DOZZLE_AGENT_PORT}:7007 + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - dozzle_agent_data:/data + environment: + - DOZZLE_HOSTNAME=${DOZZLE_HOSTNAME} + networks: + - tnet +volumes: + dozzle_data: null + dozzle_agent_data: null +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/gitea/compose.yaml +services: + server: + image: gitea/gitea:latest + container_name: gitea + environment: + - USER_UID=1000 + - USER_GID=1000 + - GITEA__database__DB_TYPE=postgres + - GITEA__database__HOST=${DB_IP}:5432 + - GITEA__database__NAME=gitea + - GITEA__database__USER=gitea + - GITEA__database__PASSWD=gitea + - GITEA__service__DISABLE_REGISTRATION=true + - GITEA__log__MODE=console + - GITEA__log__LEVEL=Info + - GITEA__log__ROUTER=console + restart: unless-stopped + volumes: + - gitea_data:/data + - /etc/timezone:/etc/timezone:ro + - /etc/localtime:/etc/localtime:ro + ports: + - 3000:3000 + - 222:22 + labels: + - homepage.group=Apps + - homepage.name=Gitea + - homepage.icon=si-gitea + - homepage.description=Git Repo (ana) + - homepage.href=https://gitea.phasefinal.com + - traefik.enable=true + - traefik.http.routers.gitea.tls=true + - traefik.http.routers.gitea.rule=Host(`gitea.phasefinal.com`) + - traefik.http.routers.gitea.tls.certresolver=anaprod + - traefik.http.services.gitea.loadbalancer.server.port=3000 + - traefik.http.routers.gitea.middlewares=crowdsec@file + - crowdsec.labels.type=gitea + networks: + - tnet +networks: + tnet: + name: traefik-net + external: true +volumes: + gitea_data: null + +>>> /opt/docker/compose/ittools/compose.yaml +services: + it-tools: + image: corentinth/it-tools:latest + container_name: it-tools + restart: unless-stopped + ports: + - "8780:80" + labels: + - homepage.group=Apps + - homepage.name=IT Tools + - homepage.icon=mdi-tools + - homepage.description=IT Dev Tools + - homepage.href=http://10.250.50.70:8780 +>>> /opt/docker/compose/mailrise/compose.yaml +#version: '3' +services: + mailrise: + image: yoryan/mailrise:latest + container_name: mailrise + labels: + - homepage.group=PFI-ANA + - homepage.name=Mailrise + - homepage.icon=mdi-mail + - homepage.description=Mail to Notification Agent + - homepage.href=http://10.250.50.70:8025 + volumes: + - /opt/docker/conf/mailrise/mailrise.conf:/etc/mailrise.conf:ro + restart: unless-stopped + environment: + - USER=1000:1000 + ports: + - 8025:8025 +networks: {} + +>>> /opt/docker/compose/mattermost/compose.yaml +#version: "2.4" + +services: + mattermost: + image: mattermost/mattermost-team-edition:latest + restart: unless-stopped + tmpfs: + - /tmp + volumes: +# - mattermost_data:/mm + - mattermost_data:/mattermost +# - /opt/docker/conf/mattermost:/mattermost/config +# - /opt/docker/data/mattermost/data:/mattermost/data +# - /opt/docker/data/mattermost/logs:/mattermost/logs +# - /opt/docker/data/mattermost/plugins:/mattermost/plugins +# - /opt/docker/data/mattermost/client/plugins:/mattermost/client/plugins +# - /opt/docker/data/mattermost/bleve-indexes:/mattermost/bleve-indexes + environment: + TZ: America/Los_Angeles + DOMAIN: mm.phasefinal.com + MM_SQLSETTINGS_DRIVERNAME: postgres + MM_SQLSETTINGS_DATASOURCE: ${DATASOURCE} + ports: + - 8065:8065 + - 8443:8443/udp + user: 1000:1000 + labels: + - homepage.group=Apps + - homepage.name=MatterMost + - homepage.icon=si-mattermost + - homepage.description=Chat Application (ana) + - homepage.href=https://mm.phasefinal.com + - traefik.enable=true + - traefik.http.routers.mattermost.tls=true + - traefik.http.routers.mattermost.rule=Host(`mm.phasefinal.com`) + - traefik.http.routers.mattermost.tls.certresolver=anaprod + - traefik.http.services.mattermost.loadbalancer.server.port=8065 + networks: + - tnet +networks: + tnet: + name: traefik-net + external: true +volumes: + mattermost_data: null + +>>> /opt/docker/compose/openwebui/compose.yaml +version: "3" +services: + open-webui: + ports: + - 3100:8080 + volumes: + - open-webui:/app/backend/data + restart: unless-stopped + image: ghcr.io/open-webui/open-webui:main + labels: + - homepage.group=AI Systems + - homepage.name=Open WebUI + - homepage.icon=mdi-chat + - homepage.description=Open WebUI Chat - ana - 3100 + - homepage.href=http://10.250.50.70:3100 + networks: + - tnet + env_file: + - .env +volumes: + open-webui: null +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/restic/compose.yaml +# ignored : docker pull restic/rest-server:latest + +#version: "3.3" +services: + rest-server: + ports: + - 8000:8000 + labels: + - homepage.group=PFI-ANA + - homepage.name=Restic + - homepage.icon=mdi-cloud-upload + - homepage.description=Restic Backup Server (8000:ana) + - homepage.href=http://10.250.50.70:8000 + volumes: + - /mnt/backup/restic/repo/ana:/data + container_name: rest_server + image: restic/rest-server + networks: + - tnet +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/rustdesk/compose.yaml +version: "3" +services: + hbbs: + container_name: hbbs + image: rustdesk/rustdesk-server:latest + command: hbbs + volumes: + - rustdesk_data:/root + #- /opt/docker/data/rustdesk:/root + network_mode: host + depends_on: + - hbbr + restart: unless-stopped + hbbr: + container_name: hbbr + image: rustdesk/rustdesk-server:latest + command: hbbr + labels: + - homepage.group=PFI-ANA + - homepage.name=Rustdesk + - homepage.icon=si-rustdesk + - homepage.description=Rustdesk Relay Server (ana) + - homepage.href=https://rustdesk.phasefinal.com + volumes: + - rustdesk_data:/root + #- /opt/docker/data/rustdesk:/root + network_mode: host + restart: unless-stopped +networks: {} +volumes: + rustdesk_data: null + +>>> /opt/docker/compose/seafile/compose.yaml +# version: '2.0' +services: + db: + image: mariadb:10.6 + container_name: seafile-mysql + environment: + - MYSQL_ROOT_PASSWORD=[REDACTED-upstream-compose-inlines-this] # Requested, set the root's password of MySQL service. + - MYSQL_LOG_CONSOLE=true + volumes: + - seafile_db:/var/lib/mysql # Requested, specifies the path to MySQL data persistent store. + networks: + - tnet + memcached: + image: memcached:1.6.18 + container_name: seafile-memcached + entrypoint: memcached -m 256 + networks: + - tnet + seafile: + image: seafileltd/seafile-mc:11.0-latest + container_name: seafile + ports: + - 9180:80 + volumes: + - seafile_datastore:/shared # Requested, specifies the path to Seafile data persistent store. + environment: + - DB_HOST=db + - DB_ROOT_PASSWD=${DB_ROOT_PW} + - TIME_ZONE=America/Los_Angeles + - SEAFILE_ADMIN_EMAIL=${SEAFILE_ADMIN_EMAIL} + - SEAFILE_ADMIN_PASSWORD=${SEAFILE_ADMIN_PW} + depends_on: + - db + - memcached + labels: + - homepage.group=Apps + - homepage.name=SeaFile + - homepage.icon=mdi-sync-circle + - homepage.description=File Sync Service (ana) + - homepage.href=https://seafile.phasefinal.com + - traefik.enable=true + - traefik.http.routers.seafile.tls=true + - traefik.http.routers.seafile.rule=Host(`seafile.phasefinal.com`) + - traefik.http.routers.seafile.tls.certresolver=anaprod + networks: + - tnet + env_file: + - .env +volumes: + seafile_db: null + seafile_datastore: null +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/searxng/compose.yaml +services: + searxng: + image: searxng/searxng:latest + container_name: searxng + restart: unless-stopped + # ------------------------------------------------------------------ + # Port binding — 9996 on all interfaces. + # Change to "127.0.0.1:9996:8080" to restrict to localhost only. + # Traefik handles public routing and TLS via the labels below. + # ------------------------------------------------------------------ + ports: + - 9996:8080 + # ------------------------------------------------------------------ + # Volumes + # Config: settings.yml bind-mounted read-only into the container. + volumes: + - /opt/docker/conf/searxng/searxng-settings.yml:/etc/searxng/settings.yml:ro + # ------------------------------------------------------------------ + # Environment — see https://docs.searxng.org/admin/settings/index.html + # SEARXNG_SECRET — required for cryptographic signing (cookies, etc.) + # BASE_URL — public URL SearXNG reports in pages/RSS/OPDS + # INSTANCE_NAME — shown in the page title / footer + # ------------------------------------------------------------------ + environment: + - SEARXNG_SECRET=${SEARXNG_SECRET} + - BASE_URL=https://searxng.pfi.local/ + - INSTANCE_NAME=SearXNG + # ------------------------------------------------------------------ + # Resource limits — tune for VM 102's available RAM/CPU + # ------------------------------------------------------------------ + deploy: + resources: + limits: + memory: 512M + cpus: "1.0" + reservations: + memory: 128M + # ------------------------------------------------------------------ + # Health check — SearXNG /healthz is the canonical liveness probe. + # ------------------------------------------------------------------ + healthcheck: + test: + - CMD + - wget + - --no-verbose + - --tries + - --spider + - http://localhost:8080/healthz + interval: 30s + timeout: 10s + retries: 3 + start_period: 15s + networks: + - tnet + labels: + # Traefik configuration — auto-discovery via Docker provider + - traefik.enable=true + - traefik.http.routers.searxng.rule=Host(`searxng.pfi.local`) + - traefik.http.routers.searxng.entrypoints=websecure + - traefik.http.routers.searxng.tls=true + - traefik.http.routers.searxng.service=searxng + - traefik.http.services.searxng.loadbalancer.server.port=8080 +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/sillytavern/compose.yaml +version: "3" +services: + sillytavern: + build: .. + container_name: sillytavern + hostname: sillytavern + image: ghcr.io/sillytavern/sillytavern:latest + ports: + - 8100:8000 + volumes: + - sillytavern_data:/home/node/app/data + - sillytavern_plugins:/home/node/app/plugins + - sillytavern_extensions:/home/node/app/public/scripts/extensions/third-party + - /opt/docker/conf/sillytavern:/home/node/app/config + # - /opt/docker/data/sillytavern/data:/home/node/app/data + # - /opt/docker/data/sillytavern/plugins:/home/node/app/plugins + # - /opt/docker/data/sillytavern/extensions:/home/node/app/public/scripts/extensions/third-party + restart: unless-stopped + labels: + - homepage.group=AI Systems + - homepage.name=Silly Tavern + - homepage.icon=mdi-chat + - homepage.description=Silly Tavern AI - ANA-Docker 8100 + - homepage.href=http://10.250.50.70:8100 + networks: + - tnet +networks: + tnet: + name: traefik-net + external: true +volumes: + sillytavern_data: null + sillytavern_plugins: null + sillytavern_extensions: null + +>>> /opt/docker/compose/synapse/compose.yaml +# ============================================================================== +# Matrix Synapse Stack — VM-102 (10.250.50.70) +# Domain: matrix.phasefinal.com +# +# Deploy: /opt/docker/compose/synapse/compose.yaml +# Config: /opt/docker/conf/synapse/homeserver.yaml +# +# Services: +# 1. synapse-db — Postgres 16 (internal only) +# 2. synapse — Matrix Synapse homeserver (port 8008) +# 3. element-web — Element Web client (port 8080) +# +# Stack uses traefik-net (tnet) for reverse proxy / TLS termination. +# Cert resolver: anaprod (matches existing VM-102 convention) +# ============================================================================== + +services: + # --------------------------------------------------------------------------- + # Postgres database for Synapse + # --------------------------------------------------------------------------- + synapse-db: + image: postgres:16-alpine + container_name: synapse-db + restart: unless-stopped + environment: + POSTGRES_DB: synapse + POSTGRES_USER: synapse + POSTGRES_PASSWORD: MOV0AHc26hHw9jDkjgIZqLb1zmY0um0v + POSTGRES_INITDB_ARGS: --lc-collate=C --lc-ctype=C --encoding=UTF8 + volumes: + - synapse-db-data:/var/lib/postgresql/data + networks: + - synapse-internal + healthcheck: + test: + - CMD-SHELL + - pg_isready -U synapse -d synapse + interval: 10s + timeout: 5s + retries: 5 + # --------------------------------------------------------------------------- + # Matrix Synapse homeserver + # --------------------------------------------------------------------------- + synapse: + image: matrixdotorg/synapse:v1.120.0 + container_name: synapse + restart: unless-stopped + depends_on: + synapse-db: + condition: service_healthy + volumes: + - /opt/docker/conf/synapse/homeserver.yaml:/data/homeserver.yaml:ro + - synapse-data:/data + - /opt/docker/conf/synapse/aipa_appservice.yaml:/conf/aipa_appservice.yaml:ro + networks: + - tnet + - synapse-internal + labels: + - traefik.enable=true + - traefik.http.routers.synapse.rule=Host(`matrix.phasefinal.com`) + - traefik.http.routers.synapse.tls=true + - traefik.http.routers.synapse.tls.certresolver=anaprod + - traefik.http.services.synapse.loadbalancer.server.port=8008 + # --------------------------------------------------------------------------- + # Element Web client + # --------------------------------------------------------------------------- + element-web: + image: vectorim/element-web:v1.11.80 + container_name: element-web + restart: unless-stopped + depends_on: + - synapse + volumes: + - /opt/docker/conf/synapse/element-config.json:/app/config.json:ro + networks: + - tnet + labels: + - traefik.enable=true + - traefik.http.routers.element.rule=Host(`chat.phasefinal.com`) + - traefik.http.routers.element.tls=true + - traefik.http.routers.element.tls.certresolver=anaprod + - traefik.http.services.element.loadbalancer.server.port=80 +networks: + synapse-internal: + driver: bridge + tnet: + name: traefik-net + external: true +# ============================================================================= +# Volumes +# ============================================================================= +volumes: + synapse-db-data: null + synapse-data: null + +>>> /opt/docker/compose/traefik/compose.yaml +services: + traefik: + image: traefik:latest + container_name: traefik + command: + - --configFile=/etc/traefik/traefik.yml + ports: + - 80:80 + - 8380:8080 + - 443:443 + labels: + - homepage.group=PFI-ANA + - homepage.name=traefik + - homepage.icon=si-traefikproxy + - homepage.sitemonitor=http://10.250.50.70:8380 + - homepage.href=http://10.250.50.70:8380 + - homepage.widget.type=traefik + - homepage.widget.url=http://10.250.50.70:8380 + - crowdsec.labels.type=traefik + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - /opt/docker/conf/traefik-ana/config:/etc/traefik + - /opt/docker/conf/traefik-ana/acme.json:/acme.json + - traefik-logs:/var/log/traefik # Important for CrowdSec + networks: + - tnet + restart: unless-stopped +networks: + tnet: + name: traefik-net + external: true +volumes: + traefik-logs: null + +>>> /opt/docker/compose/vaultwarden/compose.yaml +#version: '3' +services: + vaultwarden: + image: vaultwarden/server:latest + container_name: vaultwarden + volumes: + - vaultwarden_data:/data + - /opt/docker/conf/vaultwarden/config.json:/data/config.json + #- /opt/docker/data/vaultwarden:/data + ports: + - 9080:80 + restart: unless-stopped + labels: + - homepage.group=Apps + - homepage.name=Vaultwarden + - homepage.icon=si-bitwarden + - homepage.description=Password Vault (ana) + - homepage.sitemonitor=https://vaultwarden.phasefinal.com + - homepage.href=https://vaultwarden.phasefinal.com + - traefik.enable=true + - traefik.http.routers.vaultwarden.tls=true + - traefik.http.routers.vaultwarden.rule=Host(`vaultwarden.phasefinal.com`) + - traefik.http.routers.vaultwarden.tls.certresolver=anaprod + - crowdsec.labels.type=VaultWarden + networks: + - tnet + env_file: + - .env +networks: + tnet: + name: traefik-net + external: true +volumes: + vaultwarden_data: null + +===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) ===== + +/opt/docker/conf +/opt/docker/conf/crowdsec +/opt/docker/conf/crowdsec/acquis.d +/opt/docker/conf/crowdsec/acquis.yaml +/opt/docker/conf/crowdsec/blocklist-mirror.yaml +/opt/docker/conf/crowdsec/collections +/opt/docker/conf/crowdsec/collections/base-http-scenarios.yaml +/opt/docker/conf/crowdsec/collections/gitea.yml +/opt/docker/conf/crowdsec/collections/http-cve.yaml +/opt/docker/conf/crowdsec/collections/linux.yaml +/opt/docker/conf/crowdsec/collections/sshd.yaml +/opt/docker/conf/crowdsec/collections/traefik.yaml +/opt/docker/conf/crowdsec/collections/vaultwarden.yml +/opt/docker/conf/crowdsec/collections/whitelist-good-actors.yaml +/opt/docker/conf/crowdsec/config.yaml +/opt/docker/conf/crowdsec/console.yaml +/opt/docker/conf/crowdsec/contexts +/opt/docker/conf/crowdsec/contexts/bf_base.yaml +/opt/docker/conf/crowdsec/contexts/http_base.yaml +/opt/docker/conf/crowdsec/dev.yaml +/opt/docker/conf/crowdsec/hub +/opt/docker/conf/crowdsec/hub/collections +/opt/docker/conf/crowdsec/hub/collections/crowdsecurity +/opt/docker/conf/crowdsec/hub/collections/Dominic-Wagner +/opt/docker/conf/crowdsec/hub/collections/LePresidente +/opt/docker/conf/crowdsec/hub/contexts +/opt/docker/conf/crowdsec/hub/contexts/crowdsecurity +/opt/docker/conf/crowdsec/hub/.index.json +/opt/docker/conf/crowdsec/hub/parsers +/opt/docker/conf/crowdsec/hub/parsers/s00-raw +/opt/docker/conf/crowdsec/hub/parsers/s01-parse +/opt/docker/conf/crowdsec/hub/parsers/s02-enrich +/opt/docker/conf/crowdsec/hub/postoverflows +/opt/docker/conf/crowdsec/hub/postoverflows/s00-enrich +/opt/docker/conf/crowdsec/hub/postoverflows/s01-whitelist +/opt/docker/conf/crowdsec/hub/scenarios +/opt/docker/conf/crowdsec/hub/scenarios/crowdsecurity +/opt/docker/conf/crowdsec/hub/scenarios/Dominic-Wagner +/opt/docker/conf/crowdsec/hub/scenarios/LePresidente +/opt/docker/conf/crowdsec/hub/scenarios/ltsich +/opt/docker/conf/crowdsec/local_api_credentials.yaml +/opt/docker/conf/crowdsec/notifications +/opt/docker/conf/crowdsec/notifications/email.yaml +/opt/docker/conf/crowdsec/notifications/file.yaml +/opt/docker/conf/crowdsec/notifications/http.yaml +/opt/docker/conf/crowdsec/notifications/sentinel.yaml +/opt/docker/conf/crowdsec/notifications/slack.yaml +/opt/docker/conf/crowdsec/notifications/splunk.yaml +/opt/docker/conf/crowdsec/online_api_credentials.yaml +/opt/docker/conf/crowdsec/parsers +/opt/docker/conf/crowdsec/parsers/s00-raw +/opt/docker/conf/crowdsec/parsers/s00-raw/cri-logs.yaml +/opt/docker/conf/crowdsec/parsers/s00-raw/docker-logs.yaml +/opt/docker/conf/crowdsec/parsers/s00-raw/syslog-logs.yaml +/opt/docker/conf/crowdsec/parsers/s01-parse +/opt/docker/conf/crowdsec/parsers/s01-parse/gitea-logs.yaml +/opt/docker/conf/crowdsec/parsers/s01-parse/sshd-logs.yaml +/opt/docker/conf/crowdsec/parsers/s01-parse/sshd-success-logs.yaml +/opt/docker/conf/crowdsec/parsers/s01-parse/traefik-logs.yaml +/opt/docker/conf/crowdsec/parsers/s01-parse/vaultwarden-logs.yaml +/opt/docker/conf/crowdsec/parsers/s02-enrich +/opt/docker/conf/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml +/opt/docker/conf/crowdsec/parsers/s02-enrich/geoip-enrich.yaml +/opt/docker/conf/crowdsec/parsers/s02-enrich/http-logs.yaml +/opt/docker/conf/crowdsec/parsers/s02-enrich/public-dns-allowlist.yaml +/opt/docker/conf/crowdsec/parsers/s02-enrich/whitelists.yaml +/opt/docker/conf/crowdsec/patterns +/opt/docker/conf/crowdsec/patterns/aws +/opt/docker/conf/crowdsec/patterns/bacula +/opt/docker/conf/crowdsec/patterns/bro +/opt/docker/conf/crowdsec/patterns/cowrie_honeypot +/opt/docker/conf/crowdsec/patterns/exim +/opt/docker/conf/crowdsec/patterns/firewalls +/opt/docker/conf/crowdsec/patterns/haproxy +/opt/docker/conf/crowdsec/patterns/java +/opt/docker/conf/crowdsec/patterns/junos +/opt/docker/conf/crowdsec/patterns/linux-syslog +/opt/docker/conf/crowdsec/patterns/mcollective +/opt/docker/conf/crowdsec/patterns/modsecurity +/opt/docker/conf/crowdsec/patterns/mongodb +/opt/docker/conf/crowdsec/patterns/mysql +/opt/docker/conf/crowdsec/patterns/nagios +/opt/docker/conf/crowdsec/patterns/nginx +/opt/docker/conf/crowdsec/patterns/paths +/opt/docker/conf/crowdsec/patterns/postgresql +/opt/docker/conf/crowdsec/patterns/rails +/opt/docker/conf/crowdsec/patterns/redis +/opt/docker/conf/crowdsec/patterns/ruby +/opt/docker/conf/crowdsec/patterns/smb +/opt/docker/conf/crowdsec/patterns/ssh +/opt/docker/conf/crowdsec/patterns/tcpdump +/opt/docker/conf/crowdsec/postoverflows +/opt/docker/conf/crowdsec/postoverflows/s00-enrich +/opt/docker/conf/crowdsec/postoverflows/s00-enrich/rdns.yaml +/opt/docker/conf/crowdsec/postoverflows/s01-whitelist +/opt/docker/conf/crowdsec/postoverflows/s01-whitelist/cdn-whitelist.yaml +/opt/docker/conf/crowdsec/postoverflows/s01-whitelist/seo-bots-whitelist.yaml +/opt/docker/conf/crowdsec/profiles.yaml +/opt/docker/conf/crowdsec/scenarios +/opt/docker/conf/crowdsec/scenarios/apache_log4j2_cve-2021-44228.yaml +/opt/docker/conf/crowdsec/scenarios/CVE-2017-9841.yaml +/opt/docker/conf/crowdsec/scenarios/CVE-2019-18935.yaml +/opt/docker/conf/crowdsec/scenarios/CVE-2022-26134.yaml +/opt/docker/conf/crowdsec/scenarios/CVE-2022-35914.yaml +/opt/docker/conf/crowdsec/scenarios/CVE-2022-37042.yaml +/opt/docker/conf/crowdsec/scenarios/CVE-2022-40684.yaml +/opt/docker/conf/crowdsec/scenarios/CVE-2022-41082.yaml +/opt/docker/conf/crowdsec/scenarios/CVE-2022-41697.yaml +/opt/docker/conf/crowdsec/scenarios/CVE-2022-42889.yaml +/opt/docker/conf/crowdsec/scenarios/CVE-2022-44877.yaml +/opt/docker/conf/crowdsec/scenarios/CVE-2022-46169.yaml +/opt/docker/conf/crowdsec/scenarios/CVE-2023-22515.yaml +/opt/docker/conf/crowdsec/scenarios/CVE-2023-22518.yaml +/opt/docker/conf/crowdsec/scenarios/CVE-2023-49103.yaml +/opt/docker/conf/crowdsec/scenarios/CVE-2024-0012.yaml +/opt/docker/conf/crowdsec/scenarios/CVE-2024-38475.yaml +/opt/docker/conf/crowdsec/scenarios/CVE-2024-9474.yaml +/opt/docker/conf/crowdsec/scenarios/f5-big-ip-cve-2020-5902.yaml +/opt/docker/conf/crowdsec/scenarios/fortinet-cve-2018-13379.yaml +/opt/docker/conf/crowdsec/scenarios/gitea-bf.yaml +/opt/docker/conf/crowdsec/scenarios/gitea-scraper.yaml +/opt/docker/conf/crowdsec/scenarios/grafana-cve-2021-43798.yaml +/opt/docker/conf/crowdsec/scenarios/http-admin-interface-probing.yaml +/opt/docker/conf/crowdsec/scenarios/http-backdoors-attempts.yaml +/opt/docker/conf/crowdsec/scenarios/http-bad-user-agent.yaml +/opt/docker/conf/crowdsec/scenarios/http-crawl-non_statics.yaml +/opt/docker/conf/crowdsec/scenarios/http-cve-2021-41773.yaml +/opt/docker/conf/crowdsec/scenarios/http-cve-2021-42013.yaml +/opt/docker/conf/crowdsec/scenarios/http-cve-probing.yaml +/opt/docker/conf/crowdsec/scenarios/http-generic-bf.yaml +/opt/docker/conf/crowdsec/scenarios/http-generic-test.yaml +/opt/docker/conf/crowdsec/scenarios/http-open-proxy.yaml +/opt/docker/conf/crowdsec/scenarios/http-path-traversal-probing.yaml +/opt/docker/conf/crowdsec/scenarios/http-probing.yaml +/opt/docker/conf/crowdsec/scenarios/http-sap-interface-probing.yaml +/opt/docker/conf/crowdsec/scenarios/http-sensitive-files.yaml +/opt/docker/conf/crowdsec/scenarios/http-sqli-probing.yaml +/opt/docker/conf/crowdsec/scenarios/http-w00tw00t.yaml +/opt/docker/conf/crowdsec/scenarios/http-wordpress-scan.yaml +/opt/docker/conf/crowdsec/scenarios/http-xss-probing.yaml +/opt/docker/conf/crowdsec/scenarios/jira_cve-2021-26086.yaml +/opt/docker/conf/crowdsec/scenarios/netgear_rce.yaml +/opt/docker/conf/crowdsec/scenarios/pulse-secure-sslvpn-cve-2019-11510.yaml +/opt/docker/conf/crowdsec/scenarios/spring4shell_cve-2022-22965.yaml +/opt/docker/conf/crowdsec/scenarios/ssh-bf.yaml +/opt/docker/conf/crowdsec/scenarios/ssh-cve-2024-6387.yaml +/opt/docker/conf/crowdsec/scenarios/ssh-generic-test.yaml +/opt/docker/conf/crowdsec/scenarios/ssh-refused-conn.yaml +/opt/docker/conf/crowdsec/scenarios/ssh-slow-bf.yaml +/opt/docker/conf/crowdsec/scenarios/ssh-time-based-bf.yaml +/opt/docker/conf/crowdsec/scenarios/thinkphp-cve-2018-20062.yaml +/opt/docker/conf/crowdsec/scenarios/vaultwarden-bf.yaml +/opt/docker/conf/crowdsec/scenarios/vmware-cve-2022-22954.yaml +/opt/docker/conf/crowdsec/scenarios/vmware-vcenter-vmsa-2021-0027.yaml +/opt/docker/conf/crowdsec/simulation.yaml +/opt/docker/conf/crowdsec/user.yaml +/opt/docker/conf/mailrise +/opt/docker/conf/mailrise/mailrise.conf +/opt/docker/conf/mattermost +/opt/docker/conf/mattermost/config.json +/opt/docker/conf/searxng +/opt/docker/conf/searxng/searxng-settings.yml +/opt/docker/conf/sillytavern +/opt/docker/conf/sillytavern/config.yaml +/opt/docker/conf/synapse +/opt/docker/conf/synapse/aipa_appservice.yaml +/opt/docker/conf/synapse/element-config.json +/opt/docker/conf/synapse/homeserver.yaml +/opt/docker/conf/traefik-ana +/opt/docker/conf/traefik-ana/acme.json +/opt/docker/conf/traefik-ana/config +/opt/docker/conf/traefik-ana/config/acme.json +/opt/docker/conf/traefik-ana/config/dynamic.yml +/opt/docker/conf/traefik-ana/config/traefik.yml +/opt/docker/conf/vaultwarden +/opt/docker/conf/vaultwarden/config.json + +===== LISTENING PORTS ===== + +0.0.0.0:111 +0.0.0.0:22 +0.0.0.0:222 +0.0.0.0:3000 +0.0.0.0:3100 +0.0.0.0:41412 +0.0.0.0:443 +0.0.0.0:5001 +0.0.0.0:80 +0.0.0.0:8000 +0.0.0.0:8025 +0.0.0.0:8088 +0.0.0.0:8090 +0.0.0.0:8100 +0.0.0.0:8380 +0.0.0.0:8780 +0.0.0.0:9080 +0.0.0.0:9180 +0.0.0.0:9898 +0.0.0.0:9996 +[::]:111 +127.0.0.1:42675 +*:21115 +*:21116 +*:21117 +*:21118 +*:21119 +[::]:22 +[::]:222 +*:2375 +[::]:3000 +[::]:3100 +[::]:41412 +[::]:443 +[::]:5001 +[::]:80 +[::]:8000 +[::]:8025 +[::]:8088 +[::]:8090 +[::]:8100 +[::]:8380 +[::]:8780 +[::]:9080 +[::]:9180 +[::]:9898 +[::]:9996 + +===== MODEL / HUGGINGFACE CACHES ===== + + +===== DOCKER-ADJACENT SYSTEMD SERVICES ===== + +containerd.service running +docker.service running + +===== DONE ===== + +Paste the above back into the chat, or pass a path as argv[1] to save. diff --git a/servers/ana-ml2/README.md b/servers/ana-ml2/README.md new file mode 100644 index 0000000..45f8bd8 --- /dev/null +++ b/servers/ana-ml2/README.md @@ -0,0 +1,61 @@ +# ana-ml2 + +Primary AI inference host for PFI. + +## Network + +- **LAN IP:** 10.250.50.54 +- **SSH:** standard port 22 + +## Hardware + +- **CPU:** AMD EPYC 9254 24-core (96 threads) +- **RAM:** 566 GB +- **GPUs:** 2x NVIDIA RTX 6000 Ada Generation (46 GB VRAM each, GPU 0 and GPU 1) +- **Storage:** ZFS `zroot` (434 GB root) + `tank` pool (8.6 TB at `/tank`) +- **OS:** Debian 13 (trixie), kernel 6.12.x +- **Docker:** 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2 + +## Key paths + +| Path | Purpose | +|------|---------| +| `/opt/docker/compose//` | Compose files | +| `/opt/docker/conf//` | Config bind mounts | +| `/tank/aimodels/huggingface/` | HF cache (267 GB, pre-downloaded models) | +| `/tank/aimodels/llm/` | Legacy GGUF models (790 GB, referenced by llama-swap as `/models/`) | +| `/var/lib/docker/` | Docker data (on zroot) | + +## Running stacks + +| Stack | Port | Notes | +|-------|------|-------| +| llama-swap | 9292 | GGUF model server via llama.cpp | +| vllm-embed (Qwen3) | 8001 | OpenAI-compatible embeddings; part of the `vllm-qwen3` stack (GPU 1) | +| vllm-rerank (Qwen3) | 8002 | OpenAI-compatible reranker; part of the `vllm-qwen3` stack (GPU 1) | +| dockge | 5001 | Docker stack management UI | +| dozzle-agent | 7007 | Log agent; reports to the Dozzle hub on ana-docker | +| beszel-agent | 45876 | Metrics agent; reports to the Beszel hub on ana-docker | + +**Retired since last README update:** +- `infinity` — replaced by `vllm-qwen3` after the upstream image stopped shipping a `transformers` build that knew Qwen3. +- `LibreChat (+ rag_api, vectordb, mongodb, meilisearch)` — removed from this host. +- `searxng` — now hosted on ana-docker for the whole fleet. +- Residual networks (`librechat_default`, `kokoro-tts-gpu_default`) from prior experiments are still present; safe to `docker network rm` at leisure. + +## Refresh state + +```bash +scripts/refresh-server-info.sh ana-ml2 +``` + +Latest snapshot: `system-details.txt` (regenerate as needed). + +## GPU allocation policy + +By default, no container is pinned. For predictable performance when multiple GPU workloads run concurrently: + +- **GPU 0:** heavy LLM (llama-swap big models). +- **GPU 1:** light services (both vllm-qwen3 services share this GPU via `--gpu-memory-utilization`). + +Use `deploy.resources.reservations.devices[].device_ids: [""]` in compose to pin. diff --git a/servers/ana-ml2/ssh-target b/servers/ana-ml2/ssh-target new file mode 100644 index 0000000..d1da030 --- /dev/null +++ b/servers/ana-ml2/ssh-target @@ -0,0 +1 @@ +10.250.50.54 diff --git a/servers/ana-ml2/system-details.txt b/servers/ana-ml2/system-details.txt new file mode 100644 index 0000000..d43a031 --- /dev/null +++ b/servers/ana-ml2/system-details.txt @@ -0,0 +1,739 @@ + +===== HOST ===== + +Hostname: ana-ml2 +Date: 2026-04-19T22:15:48-07:00 +Uptime: up 33 weeks, 1 day, 14 minutes +OS: Debian GNU/Linux 13 (trixie) +Kernel: 6.12.41+deb13-amd64 +Arch: x86_64 + +===== HARDWARE ===== + +CPU cores: 96 +CPU model: AMD EPYC 9254 24-Core Processor +MemTotal: 566.6 GB +MemAvailable: 500.9 GB + +===== GPUS ===== + +index, name, memory.total [MiB], memory.free [MiB], driver_version +0, NVIDIA RTX 6000 Ada Generation, 46068 MiB, 45456 MiB, 580.65.06 +1, NVIDIA RTX 6000 Ada Generation, 46068 MiB, 34072 MiB, 580.65.06 + +===== FILESYSTEMS (df) ===== + +Filesystem Size Used Avail Use% Mounted on +zroot/ROOT/debian 434G 169G 265G 39% / +efivarfs 128K 58K 66K 47% /sys/firmware/efi/efivars +zroot/home 280G 15G 265G 6% /home +/dev/sda1 511M 92M 420M 18% /boot/efi +tank 8.6T 1.3T 7.3T 16% /tank + +===== PERSISTENT MOUNTS (/etc/fstab, non-comment) ===== + +UUID="3D9B-8E0C" /boot/efi vfat defaults 0 0 + +===== TARGETED DATA PATHS ===== + +/tank (total: 1.3T) + total 15 + drwxrwxrwx 6 root root 6 2026-04-11 23:17 . + drwxr-xr-x 18 root root 26 2026-03-29 16:08 .. + drwxrwxr-x 7 llmuser llm 8 2026-04-17 14:09 aimodels + drwxrwxr-x 3 llmuser llm 3 2025-09-08 13:21 comfy + drwxrwxr-x 4 llmuser llmuser 4 2026-04-11 23:17 kokoro + drwxrwxr-x 5 lkraven lkraven 6 2025-09-10 18:35 vibevoice + +/opt (total: 16G) + total 79 + drwxrwxrwx 13 root root 13 2026-04-17 23:00 . + drwxr-xr-x 18 root root 26 2026-03-29 16:08 .. + drwx--x--x 4 root root 4 2025-09-02 12:53 containerd + drwxrwxr-x 4 lkraven lkraven 4 2025-09-03 20:19 docker + drwxrwxr-x 8 llmuser llm 17 2026-04-17 17:50 heretic + drwxrwxr-x 15 llmuser llmuser 36 2026-04-11 23:24 Kokoro-FastAPI + drwxrwxr-x 20 llmuser llmuser 39 2025-10-09 21:42 LibreChat + drwxrwxr-x 26 llmuser llmuser 56 2025-09-04 21:01 llama.cpp + drwxrwxr-x 13 llmuser llmuser 25 2025-09-02 12:50 llama-swap + drwxrwxr-x 3 llmuser llmuser 11 2026-04-18 15:48 llmcompressor + drwxr-xr-x 4 root root 4 2025-08-30 22:54 nvidia + drwxrwxr-x 6 llmuser llmuser 15 2025-09-24 09:53 parakeet-tdt-0.6b-v2-fastapi + drwxrwxr-x 2 llmuser llmuser 6 2025-09-05 10:03 uv + +/opt/docker (total: 110M) + total 18 + drwxrwxr-x 4 lkraven lkraven 4 2025-09-03 20:19 . + drwxrwxrwx 13 root root 13 2026-04-17 23:00 .. + drwxrwxr-x 12 llmuser llm 12 2026-04-19 00:58 compose + drwxrwxr-x 3 llmuser llmuser 3 2026-04-18 22:40 conf + +/opt/docker/compose (total: 110M) + total 14 + drwxrwxr-x 12 llmuser llm 12 2026-04-19 00:58 . + drwxrwxr-x 4 lkraven lkraven 4 2025-09-03 20:19 .. + drwxr-xr-x 2 root root 4 2026-04-19 00:58 beszel-agent + drwxr-xr-x 2 llmuser llm 4 2025-09-05 20:36 comfyui + drwxrwxr-x 2 llmuser llm 3 2025-09-03 17:23 dockge + drwxr-xr-x 2 root root 4 2026-04-19 00:28 dozzle-agent + drwxrwxr-x 3 llmuser llmuser 4 2026-04-11 23:17 kokoro + drwxr-xr-x 2 llmuser llm 4 2025-09-03 17:34 llama-swap + drwxr-xr-x 2 llmuser llm 4 2025-09-24 12:32 parakeet + drwxr-xr-x 2 llmuser llmuser 2 2026-04-11 11:08 synapse + drwxr-xr-x 2 llmuser llm 4 2025-09-10 18:03 vibevoice + drwxr-xr-x 2 root root 4 2026-04-18 22:59 vllm-qwen3 + +/opt/docker/conf (total: 18K) + total 2 + drwxrwxr-x 3 llmuser llmuser 3 2026-04-18 22:40 . + drwxrwxr-x 4 lkraven lkraven 4 2025-09-03 20:19 .. + drwxrwxr-x 2 llmuser llmuser 4 2026-04-19 17:00 llama-swap + +/var/lib/docker (total: 8.5K) + +/srv (total: 512) + total 9 + drwxr-xr-x 2 root root 2 2025-08-30 18:46 . + drwxr-xr-x 18 root root 26 2026-03-29 16:08 .. + + +===== DOCKER ===== + +Server: 29.3.1 Client: 29.3.1 + +----- docker info ----- +Containers: 6 (running 6, paused 0, stopped 0) +Images: 39 +Runtimes: map[io.containerd.runc.v2:{{runc [] map[]} map[org.opencontainers.runtime-spec.features:{"ociVersionMin":"1.0.0","ociVersionMax":"1.2.1","hooks":["prestart","createRuntime","createContainer","startContainer","poststart","poststop"],"mountOptions":["async","atime","bind","defaults","dev","diratime","dirsync","exec","iversion","lazytime","loud","mand","noatime","nodev","nodiratime","noexec","noiversion","nolazytime","nomand","norelatime","nostrictatime","nosuid","nosymfollow","private","ratime","rbind","rdev","rdiratime","relatime","remount","rexec","rnoatime","rnodev","rnodiratime","rnoexec","rnorelatime","rnostrictatime","rnosuid","rnosymfollow","ro","rprivate","rrelatime","rro","rrw","rshared","rslave","rstrictatime","rsuid","rsymfollow","runbindable","rw","shared","silent","slave","strictatime","suid","symfollow","sync","tmpcopyup","unbindable"],"linux":{"namespaces":["cgroup","ipc","mount","network","pid","time","user","uts"],"capabilities":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_DAC_READ_SEARCH","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_SETGID","CAP_SETUID","CAP_SETPCAP","CAP_LINUX_IMMUTABLE","CAP_NET_BIND_SERVICE","CAP_NET_BROADCAST","CAP_NET_ADMIN","CAP_NET_RAW","CAP_IPC_LOCK","CAP_IPC_OWNER","CAP_SYS_MODULE","CAP_SYS_RAWIO","CAP_SYS_CHROOT","CAP_SYS_PTRACE","CAP_SYS_PACCT","CAP_SYS_ADMIN","CAP_SYS_BOOT","CAP_SYS_NICE","CAP_SYS_RESOURCE","CAP_SYS_TIME","CAP_SYS_TTY_CONFIG","CAP_MKNOD","CAP_LEASE","CAP_AUDIT_WRITE","CAP_AUDIT_CONTROL","CAP_SETFCAP","CAP_MAC_OVERRIDE","CAP_MAC_ADMIN","CAP_SYSLOG","CAP_WAKE_ALARM","CAP_BLOCK_SUSPEND","CAP_AUDIT_READ","CAP_PERFMON","CAP_BPF","CAP_CHECKPOINT_RESTORE"],"cgroup":{"v1":true,"v2":true,"systemd":true,"systemdUser":true,"rdma":true},"seccomp":{"enabled":true,"actions":["SCMP_ACT_ALLOW","SCMP_ACT_ERRNO","SCMP_ACT_KILL","SCMP_ACT_KILL_PROCESS","SCMP_ACT_KILL_THREAD","SCMP_ACT_LOG","SCMP_ACT_NOTIFY","SCMP_ACT_TRACE","SCMP_ACT_TRAP"],"operators":["SCMP_CMP_EQ","SCMP_CMP_GE","SCMP_CMP_GT","SCMP_CMP_LE","SCMP_CMP_LT","SCMP_CMP_MASKED_EQ","SCMP_CMP_NE"],"archs":["SCMP_ARCH_AARCH64","SCMP_ARCH_ARM","SCMP_ARCH_MIPS","SCMP_ARCH_MIPS64","SCMP_ARCH_MIPS64N32","SCMP_ARCH_MIPSEL","SCMP_ARCH_MIPSEL64","SCMP_ARCH_MIPSEL64N32","SCMP_ARCH_PPC","SCMP_ARCH_PPC64","SCMP_ARCH_PPC64LE","SCMP_ARCH_RISCV64","SCMP_ARCH_S390","SCMP_ARCH_S390X","SCMP_ARCH_X32","SCMP_ARCH_X86","SCMP_ARCH_X86_64"],"knownFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"],"supportedFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"]},"apparmor":{"enabled":true},"selinux":{"enabled":true},"intelRdt":{"enabled":true},"mountExtensions":{"idmap":{"enabled":true}}},"annotations":{"io.github.seccomp.libseccomp.version":"2.6.0","org.opencontainers.runc.checkpoint.enabled":"true","org.opencontainers.runc.commit":"v1.3.4-0-gd6d73eb8","org.opencontainers.runc.version":"1.3.4\n"},"potentiallyUnsafeConfigAnnotations":["bundle","org.systemd.property.","org.criu.config"]}]} nvidia:{{nvidia-container-runtime [] map[]} map[org.opencontainers.runtime-spec.features:{"ociVersionMin":"1.0.0","ociVersionMax":"1.2.1","hooks":["prestart","createRuntime","createContainer","startContainer","poststart","poststop"],"mountOptions":["async","atime","bind","defaults","dev","diratime","dirsync","exec","iversion","lazytime","loud","mand","noatime","nodev","nodiratime","noexec","noiversion","nolazytime","nomand","norelatime","nostrictatime","nosuid","nosymfollow","private","ratime","rbind","rdev","rdiratime","relatime","remount","rexec","rnoatime","rnodev","rnodiratime","rnoexec","rnorelatime","rnostrictatime","rnosuid","rnosymfollow","ro","rprivate","rrelatime","rro","rrw","rshared","rslave","rstrictatime","rsuid","rsymfollow","runbindable","rw","shared","silent","slave","strictatime","suid","symfollow","sync","tmpcopyup","unbindable"],"linux":{"namespaces":["cgroup","ipc","mount","network","pid","time","user","uts"],"capabilities":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_DAC_READ_SEARCH","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_SETGID","CAP_SETUID","CAP_SETPCAP","CAP_LINUX_IMMUTABLE","CAP_NET_BIND_SERVICE","CAP_NET_BROADCAST","CAP_NET_ADMIN","CAP_NET_RAW","CAP_IPC_LOCK","CAP_IPC_OWNER","CAP_SYS_MODULE","CAP_SYS_RAWIO","CAP_SYS_CHROOT","CAP_SYS_PTRACE","CAP_SYS_PACCT","CAP_SYS_ADMIN","CAP_SYS_BOOT","CAP_SYS_NICE","CAP_SYS_RESOURCE","CAP_SYS_TIME","CAP_SYS_TTY_CONFIG","CAP_MKNOD","CAP_LEASE","CAP_AUDIT_WRITE","CAP_AUDIT_CONTROL","CAP_SETFCAP","CAP_MAC_OVERRIDE","CAP_MAC_ADMIN","CAP_SYSLOG","CAP_WAKE_ALARM","CAP_BLOCK_SUSPEND","CAP_AUDIT_READ","CAP_PERFMON","CAP_BPF","CAP_CHECKPOINT_RESTORE"],"cgroup":{"v1":true,"v2":true,"systemd":true,"systemdUser":true,"rdma":true},"seccomp":{"enabled":true,"actions":["SCMP_ACT_ALLOW","SCMP_ACT_ERRNO","SCMP_ACT_KILL","SCMP_ACT_KILL_PROCESS","SCMP_ACT_KILL_THREAD","SCMP_ACT_LOG","SCMP_ACT_NOTIFY","SCMP_ACT_TRACE","SCMP_ACT_TRAP"],"operators":["SCMP_CMP_EQ","SCMP_CMP_GE","SCMP_CMP_GT","SCMP_CMP_LE","SCMP_CMP_LT","SCMP_CMP_MASKED_EQ","SCMP_CMP_NE"],"archs":["SCMP_ARCH_AARCH64","SCMP_ARCH_ARM","SCMP_ARCH_MIPS","SCMP_ARCH_MIPS64","SCMP_ARCH_MIPS64N32","SCMP_ARCH_MIPSEL","SCMP_ARCH_MIPSEL64","SCMP_ARCH_MIPSEL64N32","SCMP_ARCH_PPC","SCMP_ARCH_PPC64","SCMP_ARCH_PPC64LE","SCMP_ARCH_RISCV64","SCMP_ARCH_S390","SCMP_ARCH_S390X","SCMP_ARCH_X32","SCMP_ARCH_X86","SCMP_ARCH_X86_64"],"knownFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"],"supportedFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"]},"apparmor":{"enabled":true},"selinux":{"enabled":true},"intelRdt":{"enabled":true},"mountExtensions":{"idmap":{"enabled":true}}},"annotations":{"io.github.seccomp.libseccomp.version":"2.6.0","org.opencontainers.runc.checkpoint.enabled":"true","org.opencontainers.runc.commit":"v1.3.4-0-gd6d73eb8","org.opencontainers.runc.version":"1.3.4\n"},"potentiallyUnsafeConfigAnnotations":["bundle","org.systemd.property.","org.criu.config"]}]} runc:{{runc [] map[]} map[org.opencontainers.runtime-spec.features:{"ociVersionMin":"1.0.0","ociVersionMax":"1.2.1","hooks":["prestart","createRuntime","createContainer","startContainer","poststart","poststop"],"mountOptions":["async","atime","bind","defaults","dev","diratime","dirsync","exec","iversion","lazytime","loud","mand","noatime","nodev","nodiratime","noexec","noiversion","nolazytime","nomand","norelatime","nostrictatime","nosuid","nosymfollow","private","ratime","rbind","rdev","rdiratime","relatime","remount","rexec","rnoatime","rnodev","rnodiratime","rnoexec","rnorelatime","rnostrictatime","rnosuid","rnosymfollow","ro","rprivate","rrelatime","rro","rrw","rshared","rslave","rstrictatime","rsuid","rsymfollow","runbindable","rw","shared","silent","slave","strictatime","suid","symfollow","sync","tmpcopyup","unbindable"],"linux":{"namespaces":["cgroup","ipc","mount","network","pid","time","user","uts"],"capabilities":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_DAC_READ_SEARCH","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_SETGID","CAP_SETUID","CAP_SETPCAP","CAP_LINUX_IMMUTABLE","CAP_NET_BIND_SERVICE","CAP_NET_BROADCAST","CAP_NET_ADMIN","CAP_NET_RAW","CAP_IPC_LOCK","CAP_IPC_OWNER","CAP_SYS_MODULE","CAP_SYS_RAWIO","CAP_SYS_CHROOT","CAP_SYS_PTRACE","CAP_SYS_PACCT","CAP_SYS_ADMIN","CAP_SYS_BOOT","CAP_SYS_NICE","CAP_SYS_RESOURCE","CAP_SYS_TIME","CAP_SYS_TTY_CONFIG","CAP_MKNOD","CAP_LEASE","CAP_AUDIT_WRITE","CAP_AUDIT_CONTROL","CAP_SETFCAP","CAP_MAC_OVERRIDE","CAP_MAC_ADMIN","CAP_SYSLOG","CAP_WAKE_ALARM","CAP_BLOCK_SUSPEND","CAP_AUDIT_READ","CAP_PERFMON","CAP_BPF","CAP_CHECKPOINT_RESTORE"],"cgroup":{"v1":true,"v2":true,"systemd":true,"systemdUser":true,"rdma":true},"seccomp":{"enabled":true,"actions":["SCMP_ACT_ALLOW","SCMP_ACT_ERRNO","SCMP_ACT_KILL","SCMP_ACT_KILL_PROCESS","SCMP_ACT_KILL_THREAD","SCMP_ACT_LOG","SCMP_ACT_NOTIFY","SCMP_ACT_TRACE","SCMP_ACT_TRAP"],"operators":["SCMP_CMP_EQ","SCMP_CMP_GE","SCMP_CMP_GT","SCMP_CMP_LE","SCMP_CMP_LT","SCMP_CMP_MASKED_EQ","SCMP_CMP_NE"],"archs":["SCMP_ARCH_AARCH64","SCMP_ARCH_ARM","SCMP_ARCH_MIPS","SCMP_ARCH_MIPS64","SCMP_ARCH_MIPS64N32","SCMP_ARCH_MIPSEL","SCMP_ARCH_MIPSEL64","SCMP_ARCH_MIPSEL64N32","SCMP_ARCH_PPC","SCMP_ARCH_PPC64","SCMP_ARCH_PPC64LE","SCMP_ARCH_RISCV64","SCMP_ARCH_S390","SCMP_ARCH_S390X","SCMP_ARCH_X32","SCMP_ARCH_X86","SCMP_ARCH_X86_64"],"knownFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"],"supportedFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"]},"apparmor":{"enabled":true},"selinux":{"enabled":true},"intelRdt":{"enabled":true},"mountExtensions":{"idmap":{"enabled":true}}},"annotations":{"io.github.seccomp.libseccomp.version":"2.6.0","org.opencontainers.runc.checkpoint.enabled":"true","org.opencontainers.runc.commit":"v1.3.4-0-gd6d73eb8","org.opencontainers.runc.version":"1.3.4\n"},"potentiallyUnsafeConfigAnnotations":["bundle","org.systemd.property.","org.criu.config"]}]}] +Default runtime: runc +Storage driver: overlay2 +Root dir: /var/lib/docker +Server version: 29.3.1 + +----- running containers ----- +NAMES IMAGE STATUS PORTS +beszel-agent henrygd/beszel-agent:latest Up 21 hours +dozzle-agent amir20/dozzle:latest Up 22 hours 0.0.0.0:7007->7007/tcp, 8080/tcp +llama-swap-llama-swap-1 ghcr.io/mostlygeek/llama-swap:cuda Up 5 hours (healthy) 0.0.0.0:9292->8080/tcp, [::]:9292->8080/tcp +vllm-rerank vllm/vllm-openai:latest Up 23 hours (healthy) 0.0.0.0:8002->8000/tcp, [::]:8002->8000/tcp +vllm-embed vllm/vllm-openai:latest Up 23 hours (healthy) 0.0.0.0:8001->8000/tcp, [::]:8001->8000/tcp +dockge-dockge-1 louislam/dockge:latest Up 3 weeks (healthy) 0.0.0.0:5001->5001/tcp, [::]:5001->5001/tcp + +----- all containers ----- +NAMES IMAGE STATUS +beszel-agent henrygd/beszel-agent:latest Up 21 hours +dozzle-agent amir20/dozzle:latest Up 22 hours +llama-swap-llama-swap-1 ghcr.io/mostlygeek/llama-swap:cuda Up 5 hours (healthy) +vllm-rerank vllm/vllm-openai:latest Up 23 hours (healthy) +vllm-embed vllm/vllm-openai:latest Up 23 hours (healthy) +dockge-dockge-1 louislam/dockge:latest Up 3 weeks (healthy) + +----- networks ----- +NAME DRIVER SCOPE +bridge bridge local +host host local +kokoro-tts-gpu_default bridge local +librechat_default bridge local +llama-swap_default bridge local +none null local +traefik-net bridge local + +----- networks (external, non-default — worth knowing for compose external: true) ----- +kokoro-tts-gpu_default +librechat_default +llama-swap_default +traefik-net + +----- named volumes ----- +VOLUME NAME DRIVER +88cea17a7577848d5a34a191ac1a01acb0cd196a2792cfdae6845053ede8b02b local +beszel-agent_beszel_agent_data local +ccb997d07399630358be058be83fe6f5ee64ca307f6dc03c669d9f137644b29a local +dockge_dockge_data local +dozzle-agent_dozzle_agent_data local +librechat_pgdata2 local +parakeet_parakeet_cache local +searxng_searxng-data local + +----- compose projects currently running ----- +beszel-agent +dockge +dozzle-agent +llama-swap +vllm-qwen3 + +===== COMPOSE FILES (/opt/docker/compose/) ===== + + +>>> /opt/docker/compose/beszel-agent/compose.yaml +# Beszel — lightweight server/container monitoring. +# +# Hub: single web UI with the SQLite store. Agents: per-host metric collectors +# that the hub pulls from over SSH. +# +# Multi-host layout via compose profiles: +# COMPOSE_PROFILES=hub → hub only (ana-docker) +# COMPOSE_PROFILES=hub,agent → hub + local agent on the same host +# COMPOSE_PROFILES=agent → agent only (ana-ml2) +# +# The agent uses network_mode: host so it sees real host CPU/mem/net/disk +# counters rather than container-scoped ones — that's why it can't share +# the tnet network with the hub. +# +# All tunables live in .env — edit that, not this file. + +services: + beszel: + image: henrygd/beszel:${BESZEL_VERSION} + container_name: beszel + profiles: + - hub + restart: unless-stopped + ports: + - ${BESZEL_PORT}:8090 + volumes: + - beszel_data:/beszel_data + healthcheck: + test: + - CMD + - wget + - -qO- + - http://localhost:8090/api/health + interval: 30s + timeout: 10s + retries: 3 + start_period: 15s + networks: + - tnet + labels: + - homepage.group=PFI-ANA + - homepage.name=Beszel + - homepage.icon=mdi-chart-line + - homepage.description=Server + container monitoring + - homepage.href=http://10.250.50.70:${BESZEL_PORT} + beszel-agent: + image: henrygd/beszel-agent:${BESZEL_VERSION} + container_name: beszel-agent + profiles: + - agent + restart: unless-stopped + network_mode: host + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - beszel_agent_data:/var/lib/beszel-agent + environment: + - PORT=${BESZEL_AGENT_PORT:-45876} + - KEY=${BESZEL_HUB_KEY} + - HUB_URL=${HUB_URL} + - TOKEN=${BESZEL_TOKEN} + - EXTRA_FILESYSTEMS=${BESZEL_EXTRA_FS:-} +volumes: + beszel_data: null + beszel_agent_data: null +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/comfyui/compose.yaml +services: + comfyui: + runtime: nvidia + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: all + capabilities: + - gpu + - compute + - utility + ports: + - 8188:8188 + image: mmartial/comfyui-nvidia-docker:ubuntu24_cuda13.0-latest + networks: + - tnet + volumes: + - /tank/comfy/run:/comfy/mnt + - /tank/aimodels/img/comfy:/basedir + #user: 1001:1002 + environment: + - WANTED_UID=1001 + - WANTED_GID=1002 + - BASE_DIRECTORY=/basedir + - SECURITY_LEVEL=weak + - NVIDIA_VISIBLE_DEVICES=all + - NVIDIA_DRIVER_CAPABILITIES=all + labels: + - homepage.group=AI Systems + - homepage.name=ComfyUI + - homepage.icon=mdi-panorama-variant-outline + - homepage.description=ComfyUI Image Gen (ana-ml2) + - homepage.href=http://10.250.50.54:8188 + restart: unless-stopped +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/dockge/compose.yaml +services: + dockge: + image: louislam/dockge:latest + restart: unless-stopped + ports: + # Host Port : Container Port + - 5001:5001 + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - dockge_data:/app/data + - /opt/docker/compose:/opt/docker/compose + labels: + - homepage.group=PFI-ANA + - homepage.name=Dockge-ML2 + - homepage.icon=si-portainer + - homepage.description=Docker + - homepage.href=http://10.250.50.54:5001 + environment: + # Tell Dockge where is your stacks directory + - DOCKGE_STACKS_DIR=/opt/docker/compose + networks: + - tnet + +volumes: + dockge_data: null +networks: + tnet: + name: traefik-net + external: true + + +>>> /opt/docker/compose/dozzle-agent/compose.yaml +# Dozzle — container log viewer. +# +# Multi-host layout via compose profiles: +# COMPOSE_PROFILES=hub → runs the web UI (deploy on ana-docker) +# COMPOSE_PROFILES=agent → runs the remote agent (deploy on ana-ml2) +# +# Same compose.yaml on both servers; per-host `.env` picks the profile. +# +# All tunables live in .env — edit that, not this file. + +services: + dozzle: + image: amir20/dozzle:${DOZZLE_VERSION} + container_name: dozzle + profiles: [hub] + restart: unless-stopped + ports: + - "${DOZZLE_PORT}:8080" + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - dozzle_data:/data + environment: + - DOZZLE_HOSTNAME=${DOZZLE_HOSTNAME} + - DOZZLE_REMOTE_AGENT=${DOZZLE_REMOTE_AGENT:-} + - DOZZLE_AUTH_PROVIDER=${DOZZLE_AUTH_PROVIDER:-none} + - DOZZLE_USERNAME=${DOZZLE_USERNAME:-} + - DOZZLE_PASSWORD=${DOZZLE_PASSWORD:-} + healthcheck: + test: ["CMD", "/dozzle", "healthcheck"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 15s + networks: + - tnet + labels: + - homepage.group=PFI-ANA + - homepage.name=Dozzle + - homepage.icon=mdi-text-box-search + - homepage.description=Container logs (ana-docker + ana-ml2) + - homepage.href=http://10.250.50.70:${DOZZLE_PORT} + + dozzle-agent: + image: amir20/dozzle:${DOZZLE_VERSION} + container_name: dozzle-agent + profiles: [agent] + restart: unless-stopped + command: agent + ports: + - "${DOZZLE_AGENT_BIND:-0.0.0.0}:${DOZZLE_AGENT_PORT}:7007" + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - dozzle_agent_data:/data + environment: + - DOZZLE_HOSTNAME=${DOZZLE_HOSTNAME} + networks: + - tnet + +volumes: + dozzle_data: + dozzle_agent_data: + +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/kokoro/compose.yaml +name: kokoro-tts +services: + kokoro-tts: + container_name: kokoro-tts + build: + context: ./Kokoro-FastAPI + dockerfile: docker/gpu/Dockerfile + volumes: + - /tank/kokoro/models:/app/api/src/models + - /tank/kokoro/output:/app/output + ports: + - "8765:8880" + environment: + - PYTHONPATH=/app:/app/api + - USE_GPU=true + - PYTHONUNBUFFERED=1 + - DOWNLOAD_MODEL=false + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: all + capabilities: [gpu] + networks: + - tnet + labels: + - homepage.group=AI Systems + - homepage.name=Kokoro TTS + - homepage.icon=mdi-waveform + - homepage.description=Kokoro FastAPI TTS (OpenAI-compatible) + - homepage.href=http://10.250.50.54:8765 + restart: unless-stopped + +networks: + tnet: + name: traefik-net + external: true + + +>>> /opt/docker/compose/llama-swap/compose.yaml + services: + llama-swap: + stdin_open: true + tty: true + runtime: nvidia + volumes: + - /opt/docker/conf/llama-swap/config.yaml:/app/config.yaml + - /tank/aimodels/llm:/models + - /tank/aimodels/huggingface:/hfcache + environment: + - HF_HOME=/hfcache + - HF_HUB_CACHE=/hfcache/hub + ports: + - 9292:8080 + image: ghcr.io/mostlygeek/llama-swap:cuda + networks: + - tnet + networks: + tnet: + name: traefik-net + external: true +>>> /opt/docker/compose/parakeet/compose.yaml +services: + parakeet-stt: + image: parakeet-stt + ports: + - 8300:8000 + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: all + capabilities: + - gpu + restart: unless-stopped + volumes: + - parakeet_cache:/root/.cache + networks: + - tnet + env_file: + - .env + labels: + - homepage.group=AI Systems + - homepage.name=Parakeet + - homepage.icon=mdi-talk + - homepage.description=Parakeet STT (ana-ml2) + - homepage.href=http://10.250.50.54:8300 +networks: + tnet: + name: traefik-net + external: true +volumes: + parakeet_cache: null + +>>> /opt/docker/compose/vibevoice/compose.yaml +services: + vibevoice: + container_name: vibevoice + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: all + capabilities: + - gpu + ports: + - 8745:8745 + volumes: + - /tank/vibevoice/hf:/root/.cache/huggingface + - /tank/vibevoice/voices:/app/voices + - /tank/vibevoice/state:/var/lib/eworker + environment: + - ENABLE_1_5B=true + - ENABLE_LARGE=true + - AUTH_REQUIRED=true + - CORS_ENABLED=true + - ALLOWED_ORIGINS=* + image: eworkerinc/vibevoice:latest + networks: + - tnet + labels: + - homepage.group=AI Systems + - homepage.name=VibeVoice + - homepage.icon=mdi-chat + - homepage.description=EWorkerStudio VibeVoice + - homepage.href=http://10.250.50.54:8745 + restart: unless-stopped +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/vllm-qwen3/compose.yaml +# vLLM — Qwen3 Embedding + Reranker (one stack, two services). +# +# Replaces the unmaintained Infinity stack. vLLM runs one model per process, +# so this stack brings up two containers sharing a single GPU: +# +# vllm-embed — Qwen3-Embedding served as an OpenAI /v1/embeddings server +# vllm-rerank — Qwen3-Reranker served as a /rerank + /score server +# +# The reranker is a causal-LM checkpoint; --hf-overrides re-maps it to +# Qwen3ForSequenceClassification so vLLM's reranking endpoints work and the +# model only emits two class logits (no/yes) instead of the full 151k vocab. +# +# All tunables live in .env — edit that, not this file. +# +# Pre-download models to avoid first-run delay: +# HF_HOME=/tank/aimodels/huggingface hf download Qwen/Qwen3-Embedding-0.6B +# HF_HOME=/tank/aimodels/huggingface hf download Qwen/Qwen3-Reranker-0.6B + +services: + vllm-embed: + image: vllm/vllm-openai:${VLLM_VERSION} + container_name: vllm-embed + restart: unless-stopped + ipc: host + ports: + - ${EMBED_PORT}:8000 + volumes: + - /tank/aimodels/huggingface:/hfcache + environment: + - HF_HOME=/hfcache + - HF_HUB_CACHE=/hfcache/hub + - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-} + - VLLM_API_KEY=${API_KEY:-} + command: + - ${EMBED_MODEL} + - --served-model-name + - ${EMBED_MODEL} + - --runner + - pooling + - --host + - 0.0.0.0 + - --port + - "8000" + - --gpu-memory-utilization + - ${EMBED_GPU_MEM_UTIL} + - --max-model-len + - ${EMBED_MAX_MODEL_LEN} + - --dtype + - auto + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: + - ${GPU_ID} + capabilities: + - gpu + healthcheck: + test: + - CMD + - curl + - -f + - http://localhost:8000/health + interval: 30s + timeout: 10s + retries: 3 + start_period: 180s + networks: + - tnet + labels: + - homepage.group=AI Systems + - homepage.name=vLLM Embed (Qwen3) + - homepage.icon=mdi-vector-arrange-below + - homepage.description=Qwen3 Embedding via vLLM (ana-ml2) + - homepage.href=http://10.250.50.54:${EMBED_PORT}/docs + vllm-rerank: + image: vllm/vllm-openai:${VLLM_VERSION} + container_name: vllm-rerank + restart: unless-stopped + ipc: host + ports: + - ${RERANK_PORT}:8000 + volumes: + - /tank/aimodels/huggingface:/hfcache + environment: + - HF_HOME=/hfcache + - HF_HUB_CACHE=/hfcache/hub + - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-} + - VLLM_API_KEY=${API_KEY:-} + command: + - ${RERANK_MODEL} + - --served-model-name + - ${RERANK_MODEL} + - --runner + - pooling + - --hf-overrides + - '{"architectures":["Qwen3ForSequenceClassification"],"classifier_from_token":["no","yes"],"is_original_qwen3_reranker":true}' + - --host + - 0.0.0.0 + - --port + - "8000" + - --gpu-memory-utilization + - ${RERANK_GPU_MEM_UTIL} + - --max-model-len + - ${RERANK_MAX_MODEL_LEN} + - --dtype + - auto + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: + - ${GPU_ID} + capabilities: + - gpu + healthcheck: + test: + - CMD + - curl + - -f + - http://localhost:8000/health + interval: 30s + timeout: 10s + retries: 3 + start_period: 180s + networks: + - tnet + labels: + - homepage.group=AI Systems + - homepage.name=vLLM Rerank (Qwen3) + - homepage.icon=mdi-sort-variant + - homepage.description=Qwen3 Reranker via vLLM (ana-ml2) + - homepage.href=http://10.250.50.54:${RERANK_PORT}/docs +networks: + tnet: + name: traefik-net + external: true + +===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) ===== + +/opt/docker/conf +/opt/docker/conf/llama-swap +/opt/docker/conf/llama-swap/config.oldyaml +/opt/docker/conf/llama-swap/config.yaml + +===== LISTENING PORTS ===== + +0.0.0.0:22 +0.0.0.0:5001 +0.0.0.0:7007 +0.0.0.0:8001 +0.0.0.0:8002 +0.0.0.0:9292 +[::]:22 +*:2375 +[::]:5001 +[::]:8001 +[::]:8002 +[::]:9292 + +===== MODEL / HUGGINGFACE CACHES ===== + +/tank/aimodels/huggingface (108G) + hub entries: + CACHEDIR.TAG + datasets--HuggingFaceH4--ultrachat_200k + datasets--mlabonne--harmful_behaviors + datasets--mlabonne--harmless_alpaca + models--bartowski--Meta-Llama-3.1-8B-Instruct-GGUF + models--bartowski--NousResearch_Hermes-4-14B-GGUF + models--bartowski--TheDrummer_GLM-Steam-106B-A12B-v1-GGUF + models--bartowski--TheDrummer_Skyfall-31B-v4-GGUF + models--BeaverAI--Skyfall-R1-31B-v4a-GGUF + models--ibm-granite--granite-4.0-h-small-GGUF + models--ibm-granite--granite-4.0-h-tiny-GGUF + models--ibm-granite--granite-4.0-micro-GGUF + models--mradermacher--Daredevil-8B-abliterated-dpomix-GGUF + models--mradermacher--Qwen3-30B-A3B-abliterated-erotic-i1-GGUF + models--mradermacher--Qwen3.6-35B-A3B-abliterated-i1-GGUF + models--newsletter--VibeVoice-Large-pt + models--Qwen--Qwen2.5-0.5B-Instruct + models--Qwen--Qwen3.6-35B-A3B + models--Qwen--Qwen3-Embedding-0.6B + models--Qwen--Qwen3-Omni-30B-A3B-Instruct + models--Qwen--Qwen3-Reranker-0.6B + models--Qwen--Qwen-Image-Edit-2509 + models--unsloth--embeddinggemma-300m-GGUF + models--unsloth--gemma-4-26B-A4B-it-GGUF + models--unsloth--GLM-4.6-GGUF + models--unsloth--GLM-4.7-Flash-GGUF + models--unsloth--granite-4.0-h-micro-GGUF + models--unsloth--granite-4.0-h-small-GGUF + models--unsloth--granite-4.0-h-tiny-GGUF + models--unsloth--Kimi-K2-Instruct-0905-GGUF + +/tank/aimodels/llm (790G) + +/home/lkraven/.cache/huggingface (2.1M) + hub entries: + models--Astralyra--bge-reranker-large-Q8_0-GGUF + models--ggml-org--embeddinggemma-300M-GGUF + models--ggml-org--Qwen3-Reranker-0.6B-Q8_0-GGUF + models--jinaai--jina-reranker-v3-GGUF + models--klnstpr--bge-reranker-v2-m3-Q8_0-GGUF + models--minhtd14--jina-reranker-v2-base-multilingual-Q8_0-GGUF + models--Mungert--Qwen3-Reranker-0.6B-GGUF + models--Qwen--Qwen3-Embedding-0.6B-GGUF + models--Qwen--Qwen3-Reranker-0.6B + models--unsloth--gemma-4-26B-A4B-it + models--unsloth--gemma-4-26B-A4B-it-GGUF + models--unsloth--gemma-4-31B-it-GGUF + + +===== DOCKER-ADJACENT SYSTEMD SERVICES ===== + +containerd.service running +docker.service running + +===== DONE ===== + +Paste the above back into the chat, or pass a path as argv[1] to save. diff --git a/servers/esh-docker-vm/README.md b/servers/esh-docker-vm/README.md new file mode 100644 index 0000000..9ca599e --- /dev/null +++ b/servers/esh-docker-vm/README.md @@ -0,0 +1,76 @@ +# esh-docker-vm + +General-purpose Docker host at the **ESH home-lab site** (`esteban.net` / `10.0.0.0/8` space). VM, no GPU. Separate scope from the PFI colo work but tracked here because it's part of the same fleet. + +## Network + +- **LAN IP:** 10.0.50.45 +- **FQDN:** `esh-vm-docker.esteban.net` +- **Subnet:** `10.0.50.0/24` (inferred from macvlan config) +- **SSH:** standard port 22 +- **Traefik entrypoints:** 80/443 terminate here; DNS-01 challenge via AWS Route53; protected by CrowdSec Traefik plugin + +## Hardware + +- **CPU:** 16 vCPU (QEMU virtual — VM) +- **RAM:** 15.6 GB +- **GPUs:** none +- **Storage:** 250 GB root (ext4) + NFS mounts from `10.0.50.50` +- **OS:** Debian 12 (bookworm), kernel 6.1.x +- **Docker:** running (daemon socket at `/var/run/docker.sock`; also listens on `:2375`) + +## Key paths + +| Path | Purpose | +|------|---------| +| `/opt/docker/compose//` | Compose files (also a git repo) | +| `/opt/docker/conf//` | Config bind mounts | +| `/opt/docker/docker-bu/` | Legacy backup staging (ad-hoc) | +| `/mnt/compose` | NFS — `10.0.50.50:/mnt/compose` | +| `/mnt/backup` | NFS — `10.0.50.50:/mnt/backup` (volume-backup sidecars write here) | +| `/mnt/documents` | NFS — `10.0.50.50:/mnt/documents` (Paperless-ngx source) | +| `/mnt/books` | NFS — `10.0.50.50:/mnt/books` (Calibre library) | + +## Running stacks + +| Stack | Port(s) | Notes | +|-------|---------|-------| +| traefik | 80 / 443 / 8380 | Reverse proxy + TLS (AWS Route53 DNS-01); CrowdSec bouncer plugin | +| cloudflare-ddns-esh | — | Keeps the `esteban.net` record in sync with the dynamic WAN IP | +| adguard | 53 / 853 / 8080 / 8443 / 3000 / 784 | DNS + DoT for the ESH site | +| dockge | 5001 | Docker stack management UI | +| homepage | 5100 | Dashboard (`eshhome` / `eshhome.esteban.net`) | +| uptime-kuma | 3001 | Service uptime monitor | +| homeassistant | macvlan `10.0.50.46:8123` | Home automation (direct LAN IP via macvlan on `ens18`) | +| esphome | host net / 6052 | ESPHome firmware dashboard | +| mosquitto | 1883 | MQTT broker | +| calibre | 8082 / 8181 / 8281 | Ebook server | +| calibre-web | 8083 | Ebook web UI | +| paperless-ngx (+ redis broker + volume-backup sidecar) | 8200 | Document archive; Postgres on `10.0.50.60:5432` | +| pgadmin (+ volume-backup sidecar) | 5050 | Postgres admin UI | +| drawio | 8087 / 8447 | Diagram editor | +| dozzle-agent | 7007 | Log agent; feeds ana-docker's Dozzle hub | +| beszel-esh-vm-docker (agent) | 45876 | Metrics agent; feeds ana-docker's Beszel hub | +| portainer (+ portainer_agent) | 9443 / 8000 / 9001 | Container management UI (ad-hoc, no compose dir under `/opt/docker/compose/`) | + +## Refresh state + +```bash +scripts/refresh-server-info.sh esh-docker-vm +``` + +Latest snapshot: `system-details.txt` (regenerate as needed). + +## Cross-site monitoring + +Like `nh3-docker`, this host runs **Dozzle** and **Beszel** agents that report back to the hubs on `ana-docker`, so container logs and metrics show up alongside PFI hosts in the shared dashboards. + +## Notes + +- **Macvlan for Home Assistant** — the HA container gets its own LAN IP (`10.0.50.46`) via a macvlan network on `ens18`, avoiding NAT so multicast/mDNS for HA discovery works cleanly. +- **External Postgres** — Paperless-ngx connects to a DB running elsewhere (`10.0.50.60:5432`), not a sidecar. Paperless creds in that compose file currently look like defaults; rotate before exposing. +- **Volume backups already in place** — `paperless-ngx` and `pgadmin` include `offen/docker-volume-backup:latest` sidecars that tar named volumes to `/mnt/backup/docker/esh-vm-docker//`. When the fleet-wide restic plan lands, decide whether to subsume these or leave the per-stack sidecars alone. + +## Placement rule + +Home-lab workloads for the ESH site go here. Not part of the PFI colo topology. diff --git a/servers/esh-docker-vm/ssh-target b/servers/esh-docker-vm/ssh-target new file mode 100644 index 0000000..20642e6 --- /dev/null +++ b/servers/esh-docker-vm/ssh-target @@ -0,0 +1 @@ +10.0.50.45 diff --git a/servers/esh-docker-vm/system-details.txt b/servers/esh-docker-vm/system-details.txt new file mode 100644 index 0000000..682b029 --- /dev/null +++ b/servers/esh-docker-vm/system-details.txt @@ -0,0 +1,961 @@ + +===== HOST ===== + +Hostname: esh-vm-docker.esteban.net +Date: 2026-04-19T22:15:59-07:00 +Uptime: up 2 weeks, 6 days, 5 hours, 59 minutes +OS: Debian GNU/Linux 12 (bookworm) +Kernel: 6.1.0-41-amd64 +Arch: x86_64 + +===== HARDWARE ===== + +CPU cores: 16 +CPU model: QEMU Virtual CPU version 2.5+ +MemTotal: 15.6 GB +MemAvailable: 11.4 GB + +===== GPUS ===== + +nvidia-smi not present (no NVIDIA GPUs or driver not installed) + +===== FILESYSTEMS (df) ===== + +Filesystem Size Used Avail Use% Mounted on +/dev/sda1 250G 122G 117G 52% / +10.0.50.50:/mnt/backup 92T 26G 92T 1% /mnt/backup +10.0.50.50:/mnt/compose 1.5T 4.1G 1.5T 1% /mnt/compose +10.0.50.50:/mnt/documents 92T 0 92T 0% /mnt/documents +10.0.50.50:/mnt/books 92T 96G 92T 1% /mnt/books + +===== PERSISTENT MOUNTS (/etc/fstab, non-comment) ===== + +UUID=a2e1bc05-3afa-4803-9ab2-dfee1d5c9e6c / ext4 errors=remount-ro 0 1 +UUID=4f4fa84b-2b71-4ee5-9be4-2deb4c287d8e none swap sw 0 0 +/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0 +10.0.50.50:/mnt/compose /mnt/compose nfs defaults 0 0 +10.0.50.50:/mnt/books /mnt/books nfs defaults 0 0 +10.0.50.50:/mnt/documents /mnt/documents nfs defaults 0 0 +10.0.50.50:/mnt/backup /mnt/backup nfs defaults 0 0 + +===== TARGETED DATA PATHS ===== + +/opt (total: 62M) + total 16 + drwxr-xr-x 4 root root 4096 2025-03-18 23:59 . + drwxr-xr-x 19 root root 4096 2025-12-29 14:50 .. + drwxr-xr-x 5 lkraven lkraven 4096 2025-03-18 23:58 docker + drwxrwxrwx 6 lkraven lkraven 4096 2024-06-13 22:00 docker-bu + +/opt/docker (total: 53M) + total 32 + drwxr-xr-x 5 lkraven lkraven 4096 2025-03-18 23:58 . + drwxr-xr-x 4 root root 4096 2025-03-18 23:59 .. + drwxr-xr-x 18 lkraven lkraven 4096 2026-04-19 01:08 compose + drwxr-xr-x 7 lkraven lkraven 4096 2025-11-29 00:26 conf + drwxr-xr-x 8 lkraven lkraven 4096 2025-03-18 23:59 .git + -rw-r--r-- 1 lkraven lkraven 34 2025-03-18 23:58 .gitignore + -rw-r--r-- 1 lkraven lkraven 1059 2025-03-18 23:58 LICENSE + -rw-r--r-- 1 lkraven lkraven 60 2025-03-18 23:58 README.md + +/opt/docker/compose (total: 172K) + total 76 + drwxr-xr-x 18 lkraven lkraven 4096 2026-04-19 01:08 . + drwxr-xr-x 5 lkraven lkraven 4096 2025-03-18 23:58 .. + drwxr-xr-x 2 lkraven lkraven 4096 2025-03-18 23:58 adguard + drwxr-xr-x 2 root root 4096 2026-04-19 01:07 beszel-esh-vm-docker + drwxr-xr-x 2 lkraven lkraven 4096 2025-03-18 23:58 calibre + drwxr-xr-x 2 lkraven lkraven 4096 2025-03-18 23:58 calibre-web + drwxr-xr-x 2 root root 4096 2025-10-03 22:01 cloudflare-ddns-esh + drwxr-xr-x 2 lkraven lkraven 4096 2025-03-18 23:58 dockge + drwxr-xr-x 2 root root 4096 2026-04-19 00:30 dozzle-agent + drwxr-xr-x 2 lkraven lkraven 4096 2025-03-18 23:58 drawio + drwxr-xr-x 2 lkraven lkraven 4096 2025-03-18 23:58 esphome + drwxr-xr-x 2 lkraven lkraven 4096 2025-08-31 17:39 homeassistant + drwxr-xr-x 2 lkraven lkraven 4096 2025-05-19 22:10 homepage + drwxr-xr-x 2 root root 4096 2025-11-29 00:40 mosquitto + drwxr-xr-x 2 lkraven lkraven 4096 2025-08-28 09:37 paperless-ngx + drwxr-xr-x 2 lkraven lkraven 4096 2025-08-31 17:41 pgadmin + -rw-r--r-- 1 lkraven lkraven 25 2025-03-18 23:58 README.md + drwxr-xr-x 2 lkraven lkraven 4096 2025-05-19 22:10 traefik + drwxr-xr-x 2 lkraven lkraven 4096 2025-03-18 23:58 uptimekuma + +/opt/docker/conf (total: 50M) + total 28 + drwxr-xr-x 7 lkraven lkraven 4096 2025-11-29 00:26 . + drwxr-xr-x 5 lkraven lkraven 4096 2025-03-18 23:58 .. + drwxr-xr-x 2 nas nas 4096 2026-03-20 21:00 calibre-web + drwxr-xr-x 2 lkraven lkraven 4096 2025-10-03 22:13 cloudflare-ddns + drwxr-xr-x 5 lkraven lkraven 4096 2026-03-30 22:30 homepage + drwxr-xr-x 2 1883 1883 4096 2025-11-29 00:50 mosquitto + drwxr-xr-x 3 lkraven lkraven 4096 2025-03-18 23:58 traefik-esh + +/var/lib/docker (total: 4.0K) + +/srv (total: 20K) + total 12 + drwxr-xr-x 3 root root 4096 2024-06-13 21:52 . + drwxr-xr-x 19 root root 4096 2025-12-29 14:50 .. + drwxrwxrwx 3 root root 4096 2024-06-13 21:54 backups + + +===== DOCKER ===== + +Server: 20.10.24+dfsg1 Client: 20.10.24+dfsg1 + +----- docker info ----- +Containers: 24 (running 21, paused 0, stopped 3) +Images: 84 +Runtimes: map[io.containerd.runc.v2:{runc [] } io.containerd.runtime.v1.linux:{runc [] } runc:{runc [] }] +Default runtime: runc +Storage driver: overlay2 +Root dir: /var/lib/docker +Server version: 20.10.24+dfsg1 + +----- running containers ----- +NAMES IMAGE STATUS PORTS +beszel-agent henrygd/beszel-agent:latest Up 21 hours +dozzle-agent amir20/dozzle:latest Up 22 hours 0.0.0.0:7007->7007/tcp, 8080/tcp +traefik traefik:latest Up 3 days 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:8380->8080/tcp, :::8380->8080/tcp +paperless-ngx-webserver-1 ghcr.io/paperless-ngx/paperless-ngx:latest Up 2 weeks (healthy) 0.0.0.0:8200->8000/tcp, :::8200->8000/tcp +paperless-ngx-broker-1 redis:7 Up 2 weeks 6379/tcp +paperless-ngx-backup-1 offen/docker-volume-backup:latest Up 2 weeks +homepage ghcr.io/gethomepage/homepage:latest Up 2 weeks (healthy) 0.0.0.0:5100->3000/tcp, :::5100->3000/tcp +homeassistant homeassistant/home-assistant:latest Up 30 hours +mosquitto eclipse-mosquitto:latest Up 2 weeks 0.0.0.0:1883->1883/tcp, :::1883->1883/tcp +adguardhome adguard/adguardhome Up 2 weeks 67-68/udp, 0.0.0.0:53->53/udp, :::53->53/udp, 443/udp, 0.0.0.0:53->53/tcp, :::53->53/tcp, 853/udp, 0.0.0.0:853->853/tcp, :::853->853/tcp, 3000/udp, 5443/tcp, 0.0.0.0:3000->3000/tcp, 0.0.0.0:784->784/udp, :::3000->3000/tcp, :::784->784/udp, 5443/udp, 6060/tcp, 0.0.0.0:8080->80/tcp, :::8080->80/tcp, 0.0.0.0:8443->443/tcp, :::8443->443/tcp +cloudflare-ddns timothyjmiller/cloudflare-ddns:latest Up 2 weeks +pgadmin4_container dpage/pgadmin4 Up 2 weeks 443/tcp, 0.0.0.0:5050->80/tcp, :::5050->80/tcp +pgadmin-backup-1 1127ad194f2f Up 2 weeks +esphome ghcr.io/esphome/esphome Up 2 weeks (healthy) +drawio jgraph/drawio Up 2 weeks (healthy) 0.0.0.0:8087->8080/tcp, :::8087->8080/tcp, 0.0.0.0:8447->8443/tcp, :::8447->8443/tcp +calibre-web lscr.io/linuxserver/calibre-web:latest Up 2 weeks 0.0.0.0:8083->8083/tcp, :::8083->8083/tcp +calibre lscr.io/linuxserver/calibre:latest Up 2 weeks 3000-3001/tcp, 0.0.0.0:8181->8181/tcp, :::8181->8181/tcp, 0.0.0.0:8082->8080/tcp, :::8082->8080/tcp, 0.0.0.0:8281->8081/tcp, :::8281->8081/tcp +45d2522a8cb6_uptime-kuma louislam/uptime-kuma:latest Up 2 weeks (healthy) 0.0.0.0:3001->3001/tcp, :::3001->3001/tcp +dockge-dockge-1 louislam/dockge:latest Up 2 weeks (healthy) 0.0.0.0:5001->5001/tcp, :::5001->5001/tcp +portainer_agent portainer/agent:2.19.4 Up 2 weeks 0.0.0.0:9001->9001/tcp, :::9001->9001/tcp +portainer portainer/portainer-ce:latest Up 2 weeks 0.0.0.0:8000->8000/tcp, :::8000->8000/tcp, 0.0.0.0:9443->9443/tcp, :::9443->9443/tcp, 9000/tcp + +----- all containers ----- +NAMES IMAGE STATUS +beszel-agent henrygd/beszel-agent:latest Up 21 hours +dozzle-agent amir20/dozzle:latest Up 22 hours +traefik traefik:latest Up 3 days +paperless-ngx-webserver-1 ghcr.io/paperless-ngx/paperless-ngx:latest Up 2 weeks (healthy) +paperless-ngx-broker-1 redis:7 Up 2 weeks +paperless-ngx-backup-1 offen/docker-volume-backup:latest Up 2 weeks +homepage ghcr.io/gethomepage/homepage:latest Up 2 weeks (healthy) +homeassistant homeassistant/home-assistant:latest Up 30 hours +mosquitto eclipse-mosquitto:latest Up 2 weeks +adguardhome adguard/adguardhome Up 2 weeks +friendly_maxwell 0745ced90756 Exited (13) 4 months ago +focused_pascal 0745ced90756 Exited (13) 4 months ago +pedantic_proskuriakova 0745ced90756 Exited (3) 4 months ago +cloudflare-ddns timothyjmiller/cloudflare-ddns:latest Up 2 weeks +pgadmin4_container dpage/pgadmin4 Up 2 weeks +pgadmin-backup-1 1127ad194f2f Up 2 weeks +esphome ghcr.io/esphome/esphome Up 2 weeks (healthy) +drawio jgraph/drawio Up 2 weeks (healthy) +calibre-web lscr.io/linuxserver/calibre-web:latest Up 2 weeks +calibre lscr.io/linuxserver/calibre:latest Up 2 weeks +45d2522a8cb6_uptime-kuma louislam/uptime-kuma:latest Up 2 weeks (healthy) +dockge-dockge-1 louislam/dockge:latest Up 2 weeks (healthy) +portainer_agent portainer/agent:2.19.4 Up 2 weeks +portainer portainer/portainer-ce:latest Up 2 weeks + +----- networks ----- +NAME DRIVER SCOPE +adguard_default bridge local +bridge bridge local +calibre-web_default bridge local +cloudflare-ddns-esh_default bridge local +homeassistant_macvlan_net macvlan local +homepage_default bridge local +host host local +none null local +pgadmin_default bridge local +traefik-net bridge local + +----- networks (external, non-default — worth knowing for compose external: true) ----- +adguard_default +calibre-web_default +cloudflare-ddns-esh_default +homepage_default +pgadmin_default +traefik-net + +----- named volumes ----- +VOLUME NAME DRIVER +6cdde6cf4b58151d434e8a3da23e77d9c04acdaa574a91ff7bbe60ea113a2adf local +6f7206c067f22a53bf7d0c737f74075f849c11fd535b884c4d3cbb73d275bc21 local +007fc81bd8008b22173f8e4a365b2ca43a37080902ab961a197dde0d8782c786 local +8b1ff384b6cb4859c64a21b51d32780027a3682dc1852a36617b59870ea15e32 local +adguard_adguard-confdir local +adguard_adguard-workdir local +ae127367f71c43b662b16c8232a135c59f2a873dcdf606dfa01967fad1ac915e local +beszel-esh-vm-docker_beszel_agent_data local +beszel-vm-esh-nas_beszel_agent_data local +c7a5978e1b811c4d5b15404e405c0ec32a541f8a921a054b0f725a3010a76684 local +dockge_dockge_data local +dozzle-agent_dozzle_agent_data local +homeassistant_homeassistant_data local +mosquitto_mosquitto_data local +mosquitto_mosquitto_log local +paperless-ngx_data local +paperless-ngx_media local +paperless-ngx_redisdata local +pgadmin_pgadmin-data local +portainer_data local +uptimekuma_uptime-kuma local + +----- compose projects currently running ----- +adguard +beszel-esh-vm-docker +calibre +calibre-web +cloudflare-ddns-esh +dockge +dozzle-agent +drawio +esphome +homeassistant +homepage +mosquitto +paperless-ngx +pgadmin +traefik +uptimekuma + +===== COMPOSE FILES (/opt/docker/compose/) ===== + + +>>> /opt/docker/compose/adguard/docker-compose.yml +#version: "3" +services: + adguardhome: + image: adguard/adguardhome + container_name: adguardhome + ports: + - 53:53/tcp + - 53:53/udp + - 784:784/udp + - 853:853/tcp + - 3000:3000/tcp + - 8080:80/tcp + - 8443:443/tcp + volumes: + - adguard-workdir:/opt/adguardhome/work + - adguard-confdir:/opt/adguardhome/conf + restart: unless-stopped + labels: + # This means the container will be stopped during backup to ensure + # backup integrity. You can omit this label if stopping during backup + # not required. Can be omitted if false. + - docker-volume-backup.stop-during-backup=false + - homepage.group=ESH + - homepage.name=AdGuard Home + - homepage.icon=si-adguard + - homepage.description=DNS + - homepage.href=http://10.0.50.45:8080 + - homepage.widget.type=adguard + - homepage.widget.url=http://10.0.50.45:8080 + - homepage.widget.username=lkraven + - homepage.widget.password=[REDACTED-upstream-compose-inlines-this] + networks: + - tnet + +volumes: + adguard-workdir: null + adguard-confdir: null +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/beszel-esh-vm-docker/compose.yaml +# Beszel — lightweight server/container monitoring. +# +# Hub: single web UI with the SQLite store. Agents: per-host metric collectors +# that the hub pulls from over SSH. +# +# Multi-host layout via compose profiles: +# COMPOSE_PROFILES=hub → hub only (ana-docker) +# COMPOSE_PROFILES=hub,agent → hub + local agent on the same host +# COMPOSE_PROFILES=agent → agent only (ana-ml2) +# +# The agent uses network_mode: host so it sees real host CPU/mem/net/disk +# counters rather than container-scoped ones — that's why it can't share +# the tnet network with the hub. +# +# All tunables live in .env — edit that, not this file. + +services: + beszel: + image: henrygd/beszel:${BESZEL_VERSION} + container_name: beszel + profiles: [hub] + restart: unless-stopped + ports: + - "${BESZEL_PORT}:8090" + volumes: + - beszel_data:/beszel_data + healthcheck: + test: ["CMD", "wget", "-qO-", "http://localhost:8090/api/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 15s + networks: + - tnet + labels: + - homepage.group=PFI-ANA + - homepage.name=Beszel + - homepage.icon=mdi-chart-line + - homepage.description=Server + container monitoring + - homepage.href=http://10.250.50.70:${BESZEL_PORT} + + beszel-agent: + image: henrygd/beszel-agent:${BESZEL_VERSION} + container_name: beszel-agent + profiles: [agent] + restart: unless-stopped + network_mode: host + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - beszel_agent_data:/var/lib/beszel-agent + environment: + - PORT=${BESZEL_AGENT_PORT:-45876} + - KEY=${BESZEL_HUB_KEY} + - HUB_URL=${HUB_URL} + - TOKEN=${BESZEL_TOKEN} + - EXTRA_FILESYSTEMS=${BESZEL_EXTRA_FS:-} + +volumes: + beszel_data: + beszel_agent_data: + +networks: + tnet: + name: traefik-net + external: true +>>> /opt/docker/compose/calibre/compose.yaml +version: "2.1" +services: + calibre: + image: lscr.io/linuxserver/calibre:latest + container_name: calibre + security_opt: + - seccomp:unconfined #optional + environment: + - PUID=2000 + - PGID=2000 + - TZ=America/Los_Angeles + - PASSWORD= #optional + - CLI_ARGS= #optional + volumes: + - /mnt/books/calibre:/config + ports: + - 8082:8080 + - 8181:8181 + - 8281:8081 + restart: unless-stopped + labels: + - homepage.group=Media + - homepage.name=Calibre + - homepage.icon=mdi-bookshelf + - homepage.description=EBook Server (esh) + - homepage.href=http://10.0.50.45:8082 + networks: + - tnet +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/calibre-web/compose.yaml +services: + calibre-web: + image: lscr.io/linuxserver/calibre-web:latest + container_name: calibre-web + environment: + - PUID=2000 + - PGID=2000 + - TZ=Etc/UTC + - DOCKER_MODS=linuxserver/mods:universal-calibre #optional + - OAUTHLIB_RELAX_TOKEN_SCOPE=1 #optional + volumes: + - /opt/docker/conf/calibre-web:/config + - /mnt/books/calibre/calibre_library:/books + labels: + - homepage.group=Media + - homepage.name=Calibre-Web + - homepage.icon=si-calibreweb + - homepage.description=EBook Server (esh) + - homepage.href=http://10.0.50.45:8083 + ports: + - 8083:8083 + restart: unless-stopped +networks: {} + +>>> /opt/docker/compose/cloudflare-ddns-esh/compose.yaml +services: + cloudflare-ddns: + image: timothyjmiller/cloudflare-ddns:latest + container_name: cloudflare-ddns + volumes: + - /opt/docker/conf/cloudflare-ddns/config.json:/config.json + restart: unless-stopped +networks: {} + +>>> /opt/docker/compose/dockge/compose.yaml +services: + dockge: + image: louislam/dockge:latest + restart: unless-stopped + ports: + # Host Port : Container Port + - 5001:5001 + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - dockge_data:/app/data + - /opt/docker/compose:/opt/docker/compose + labels: + - homepage.group=ESH + - homepage.name=Dockge + - homepage.icon=si-portainer + - homepage.description=Docker + - homepage.href=http://10.0.50.45:5001 + environment: + # Tell Dockge where is your stacks directory + - DOCKGE_STACKS_DIR=/opt/docker/compose + networks: + - tnet + +volumes: + dockge_data: null +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/dozzle-agent/compose.yaml +# Dozzle — container log viewer. +# +# Multi-host layout via compose profiles: +# COMPOSE_PROFILES=hub → runs the web UI (deploy on ana-docker) +# COMPOSE_PROFILES=agent → runs the remote agent (deploy on ana-ml2) +# +# Same compose.yaml on both servers; per-host `.env` picks the profile. +# +# All tunables live in .env — edit that, not this file. + +services: + dozzle: + image: amir20/dozzle:${DOZZLE_VERSION} + container_name: dozzle + profiles: + - hub + restart: unless-stopped + ports: + - ${DOZZLE_PORT}:8080 + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - dozzle_data:/data + environment: + - DOZZLE_HOSTNAME=${DOZZLE_HOSTNAME} + - DOZZLE_REMOTE_AGENT=${DOZZLE_REMOTE_AGENT:-} + - DOZZLE_AUTH_PROVIDER=${DOZZLE_AUTH_PROVIDER:-none} + - DOZZLE_USERNAME=${DOZZLE_USERNAME:-} + - DOZZLE_PASSWORD=${DOZZLE_PASSWORD:-} + healthcheck: + test: + - CMD + - /dozzle + - healthcheck + interval: 30s + timeout: 10s + retries: 3 + start_period: 15s + networks: + - tnet + labels: + - homepage.group=PFI-ANA + - homepage.name=Dozzle + - homepage.icon=mdi-text-box-search + - homepage.description=Container logs (ana-docker + ana-ml2) + - homepage.href=http://10.250.50.70:${DOZZLE_PORT} + dozzle-agent: + image: amir20/dozzle:${DOZZLE_VERSION} + container_name: dozzle-agent + profiles: + - agent + restart: unless-stopped + command: agent + ports: + - ${DOZZLE_AGENT_BIND:-0.0.0.0}:${DOZZLE_AGENT_PORT}:7007 + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - dozzle_agent_data:/data + environment: + - DOZZLE_HOSTNAME=${DOZZLE_HOSTNAME} + networks: + - tnet +volumes: + dozzle_data: null + dozzle_agent_data: null +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/drawio/compose.yaml +services: + drawio: + image: jgraph/drawio + container_name: drawio + restart: unless-stopped + ports: + - 8087:8080 + - 8447:8443 + healthcheck: + test: + - CMD-SHELL + - curl -f http://127.0.0.1:8080 || exit 1 + interval: 1m30s + timeout: 10s + retries: 5 + start_period: 10s + labels: + - homepage.group=Apps + - homepage.name=draw.io + - homepage.icon=mdi-pencil + - homepage.description=Draw.IO Graphing (esh) + - homepage.href=http://10.0.50.45:8087 + networks: + - tnet +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/esphome/compose.yaml +version: '3' +services: + esphome: + container_name: esphome + image: ghcr.io/esphome/esphome + volumes: + - /path/to/esphome/config:/config + - /etc/localtime:/etc/localtime:ro + restart: always + privileged: true + network_mode: host + environment: + - USERNAME=test + - PASSWORD=ChangeMe +>>> /opt/docker/compose/homeassistant/compose.yaml +services: + homeassistant: + image: homeassistant/home-assistant:latest + container_name: homeassistant + environment: + - PUID=2000 + - PGID=2000 + - TZ=America/Los_Angeles + volumes: + - homeassistant_data:/config + restart: unless-stopped + env_file: + - .env + networks: + macvlan_net: + ipv4_address: 10.0.50.46 + labels: + - homepage.group=Apps + - homepage.name=Home Assistant + - homepage.icon=si-homeassistant + - homepage.description=Home Automation (esh) + - homepage.href=http://10.0.50.46:8123 +networks: + macvlan_net: + driver: macvlan + driver_opts: + parent: ens18 + ipam: + config: + - subnet: 10.0.50.0/24 +volumes: + homeassistant_data: null + +>>> /opt/docker/compose/homepage/compose.yaml +services: + homepage: + image: ghcr.io/gethomepage/homepage:latest + container_name: homepage + environment: + PUID: 1000 + PGID: 1000 + ports: + - 5100:3000 + volumes: + - /opt/docker/conf/homepage:/app/config # Make sure your local config directory exists + - /var/run/docker.sock:/var/run/docker.sock:ro # optional, for docker integrations + restart: unless-stopped + networks: + - tnet + labels: + - traefik.enable=true + - traefik.http.routers.homepage.rule=host(`eshhome`) || host(`10.0.50.45`) + || host (`eshhome.esteban.net`) + - traefik.http.services.homepage.loadbalancer.server.port=3000 + - traefik.http.routers.homepage.priority=1 + env_file: + - .env +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/mosquitto/compose.yaml +services: + mosquitto: + container_name: mosquitto + image: eclipse-mosquitto:latest + restart: unless-stopped + ports: + - 1883:1883/tcp + volumes: + - /opt/docker/conf/mosquitto:/mosquitto/config + - mosquitto_data:/mosquitto/data + - mosquitto_log:/mosquitto/log + networks: + - tnet + labels: + - homepage.group=Apps + - homepage.name=Mosquitto + - homepage.icon=mdi-bug + - homepage.description=MQQT Broker port 1883 +networks: + tnet: + name: traefik-net + external: true +volumes: + mosquitto_data: null + mosquitto_log: null + +>>> /opt/docker/compose/paperless-ngx/compose.yaml +#version: "3.4" +services: + broker: + image: docker.io/library/redis:7 + restart: unless-stopped + volumes: + - redisdata:/data + networks: + - tnet + webserver: + image: ghcr.io/paperless-ngx/paperless-ngx:latest + restart: unless-stopped + depends_on: + - broker + ports: + - 8200:8000 + healthcheck: + test: + - CMD + - curl + - -fs + - -S + - --max-time + - "2" + - http://localhost:8000 + interval: 30s + timeout: 10s + retries: 5 + volumes: + - data:/usr/src/paperless/data + - media:/usr/src/paperless/media + - /mnt/documents/paperless/export:/usr/src/paperless/export + - /mnt/documents/paperless/consume:/usr/src/paperless/consume + # env_file: docker-compose.env + environment: + PAPERLESS_REDIS: redis://broker:6379 + PAPERLESS_DBHOST: 10.0.50.60 + PAPERLESS_DBPORT: 5432 + PAPERLESS_DBNAME: paperless-ng + PAPERLESS_DBUSER: paperless-ng + PAPERLESS_DBPASS: paperless-ng + # PAPERLESS_MEDIA_ROOT=/data/media + PAPERLESS_CONSUMPTION_DIR: /usr/src/paperless/consume + PAPERLESS_EXPORT_DIR: /usr/src/paperless/export + PAPERLESS_CONSUMER_POLLING: 30 + # PAPERLESS_DATA_DIR=/config + labels: + - homepage.group=Media + - homepage.name=PaperlessNGX + - homepage.icon=mdi-file-cabinet + - homepage.description=Document Store (esh) + - homepage.href=http://10.0.50.45:8200 + - homepage.sitemonitor=http://10.0.50.45:8200 + - homepage.widget.type=paperlessngx + - homepage.widget.url=http://10.0.50.45:8200 + - homepage.widget.key=${API_KEY} + networks: + - tnet + env_file: + - .env + backup: + image: offen/docker-volume-backup:latest + restart: always + volumes: + # volumes to backup. + - data:/backup/paperless-data:ro + - media:/backup/paperless-media:ro + - redisdata:/backup/paperless-redisdata:ro + - /mnt/backup/docker/esh-vm-docker/paperless:/archive + # can omit below if not stopping service. + # - /var/run/docker.sock:/var/run/docker.sock:ro + env_file: + - .env + networks: + - tnet +networks: + tnet: + name: traefik-net + external: true +volumes: + data: null + media: null + redisdata: null + +>>> /opt/docker/compose/pgadmin/compose.yaml +# version: "3.8" +services: + pgadmin: + image: dpage/pgadmin4 + container_name: pgadmin4_container + restart: always + ports: + - 5050:80 + environment: + PGADMIN_DEFAULT_EMAIL: ${DEFAULT_EMAIL} + PGADMIN_DEFAULT_PASSWORD: ${DEFAULT_PASSWORD} + volumes: + - pgadmin-data:/var/lib/pgadmin + labels: + - homepage.group=Apps + - homepage.name=PGAdmin + - homepage.icon=mdi-database + - homepage.description=pgsql Manager (esh) + - homepage.href=http://10.0.50.45:5050 + networks: + - tnet + env_file: + - .env + backup: + image: offen/docker-volume-backup:latest + restart: always + volumes: + # volumes to backup. + - pgadmin-data:/backup/pgadmin-data:ro + - /mnt/backup/docker/esh-vm-docker/pgadmin:/archive + # can omit below if not stopping service. + # - /var/run/docker.sock:/var/run/docker.sock:ro + env_file: + - .env +networks: + tnet: + name: traefik-net + external: true +volumes: + pgadmin-data: null + +>>> /opt/docker/compose/traefik/docker-compose.yml +#version: "3.3" + +services: + traefik: + image: traefik:latest + container_name: traefik + command: + - --log.level=DEBUG + - --configFile=/etc/traefik/traefik.yml + - --api.insecure=true + - --providers.docker=true + - --providers.docker.exposedbydefault=false + - --entrypoints.web.address=:80 + - --experimental.plugins.bouncer.modulename=github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin + - --experimental.plugins.bouncer.version=v1.6.0 + environment: + - AWS_ACCESS_KEY_ID=${AWS_KEY} + - AWS_SECRET_ACCESS_KEY=${AWS_SECRET} + - AWS_REGION=${AWS_REGION} + - AWS_HOSTED_ZONE_ID=${AWS_ZONEID} + ports: + - 80:80 + - 8380:8080 + - 443:443 + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - /opt/docker/conf/traefik-esh:/etc/traefik + labels: + - homepage.group=ESH + - homepage.name=traefik + - homepage.icon=si-traefikproxy + - homepage.sitemonitor=http://10.0.50.45:8380 + - homepage.href=http://10.0.50.45:8380 + - homepage.widget.type=traefik + - homepage.widget.url=http://10.0.50.45:8380 + - traefik.http.middlewares.crowdsec.plugin.bouncer.enabled=true + - traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseclapikey=${CROWDSEC_KEY} + - traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseclapihost=crowdsec:8080 + - traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseclapischeme=http + - traefik.http.middlewares.crowdsec.plugin.bouncer.crowdsecMode=live + - traefik.http.middlewares.crowdsec.plugin.bouncer.defaultDecisionSeconds=60 + networks: + - tnet + env_file: + - .env +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/uptimekuma/compose.yaml +services: + uptime-kuma: + restart: always + ports: + - 3001:3001 + volumes: + - uptime-kuma:/app/data + container_name: uptime-kuma + image: louislam/uptime-kuma:latest + labels: + - homepage.group=Apps + - homepage.name=Uptime Kuma + - homepage.icon=mdi-arrow-up-bold-circle + - homepage.description=Service Monitoring (esh) + - homepage.href=http://10.0.50.45:3001 + - homepage.widget.type=uptimekuma + - homepage.widget.url=http://10.0.50.45:3001 + - homepage.widget.slug=nethealth + networks: + - tnet +volumes: + uptime-kuma: {} +networks: + tnet: + name: traefik-net + external: true + +===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) ===== + +/opt/docker/conf +/opt/docker/conf/calibre-web +/opt/docker/conf/calibre-web/app.db +/opt/docker/conf/calibre-web/.CALIBRE_RELEASE +/opt/docker/conf/calibre-web/calibre-web.log +/opt/docker/conf/calibre-web/calibre-web.log.1 +/opt/docker/conf/calibre-web/calibre-web.log.2 +/opt/docker/conf/calibre-web/client_secrets.json +/opt/docker/conf/calibre-web/gdrive.db +/opt/docker/conf/calibre-web/.key +/opt/docker/conf/cloudflare-ddns +/opt/docker/conf/cloudflare-ddns/config.json +/opt/docker/conf/homepage +/opt/docker/conf/homepage/bookmarks.yaml +/opt/docker/conf/homepage/custom.css +/opt/docker/conf/homepage/custom.js +/opt/docker/conf/homepage/docker.yaml +/opt/docker/conf/homepage/imgs +/opt/docker/conf/homepage/imgs/bg1.jpg +/opt/docker/conf/homepage/imgs/Thumbs.db +/opt/docker/conf/homepage/kubernetes.yaml +/opt/docker/conf/homepage/logs +/opt/docker/conf/homepage/logs/homepage.log +/opt/docker/conf/homepage/nas-tls +/opt/docker/conf/homepage/nas-tls/ca.pem +/opt/docker/conf/homepage/nas-tls/cert.pem +/opt/docker/conf/homepage/nas-tls/key.pem +/opt/docker/conf/homepage/proxmox.yaml +/opt/docker/conf/homepage/services.yaml +/opt/docker/conf/homepage/settings.yaml +/opt/docker/conf/homepage/widgets.yaml +/opt/docker/conf/mosquitto +/opt/docker/conf/mosquitto/mosquitto.conf +/opt/docker/conf/mosquitto/mosquitto.log +/opt/docker/conf/mosquitto/mosquitto.passwd +/opt/docker/conf/traefik-esh +/opt/docker/conf/traefik-esh/certs +/opt/docker/conf/traefik-esh/certs/acme.json +/opt/docker/conf/traefik-esh/traefik.yml + +===== LISTENING PORTS ===== + +0.0.0.0:111 +0.0.0.0:1883 +0.0.0.0:22 +0.0.0.0:3000 +0.0.0.0:3001 +0.0.0.0:443 +0.0.0.0:5001 +0.0.0.0:5050 +0.0.0.0:5100 +0.0.0.0:53 +0.0.0.0:6052 +0.0.0.0:7007 +0.0.0.0:80 +0.0.0.0:8000 +0.0.0.0:8080 +0.0.0.0:8082 +0.0.0.0:8083 +0.0.0.0:8087 +0.0.0.0:8181 +0.0.0.0:8200 +0.0.0.0:8281 +0.0.0.0:8380 +0.0.0.0:8443 +0.0.0.0:8447 +0.0.0.0:853 +0.0.0.0:9001 +0.0.0.0:9443 +[::]:111 +127.0.0.1:35469 +[::]:1883 +[::]:22 +*:2375 +[::]:3000 +[::]:3001 +[::]:443 +[::]:5001 +[::]:5050 +[::]:5100 +[::]:53 +[::]:80 +[::]:8000 +[::]:8080 +[::]:8082 +[::]:8083 +[::]:8087 +[::]:8181 +[::]:8200 +[::]:8281 +[::]:8380 +[::]:8443 +[::]:8447 +[::]:853 +[::]:9001 +[::]:9443 + +===== MODEL / HUGGINGFACE CACHES ===== + + +===== DOCKER-ADJACENT SYSTEMD SERVICES ===== + +containerd.service running +docker.service running + +===== DONE ===== + +Paste the above back into the chat, or pass a path as argv[1] to save. diff --git a/servers/nh3-docker/README.md b/servers/nh3-docker/README.md new file mode 100644 index 0000000..0334163 --- /dev/null +++ b/servers/nh3-docker/README.md @@ -0,0 +1,52 @@ +# nh3-docker + +General-purpose Docker host for the New Hampshire (nh3) site. Small VM, no GPU. Separate LAN from the Anaheim colo. + +## Network + +- **LAN IP:** 10.100.50.40 +- **LAN subnet:** 10.100.0.0/16 (NH site) +- **FQDN:** `nh3-docker.phasefinal.com` +- **SSH:** standard port 22 + +## Hardware + +- **CPU:** 8 vCPU (QEMU virtual — VM) +- **RAM:** 7.8 GB +- **GPUs:** none +- **Storage:** 125 GB root (ext4) + NFS mounts from `10.100.50.50` (Synology) +- **OS:** Debian 12 (bookworm), kernel 6.1.x +- **Docker:** running (daemon socket at `/var/run/docker.sock`; also listens on `:2375` — be aware if firewalling) + +## Key paths + +| Path | Purpose | +|------|---------| +| `/opt/docker/compose//` | Compose files (also a git repo) | +| `/opt/docker/conf//` | Config bind mounts (empty as of last inspection) | +| `/mnt/compose` | NFS — `10.100.50.50:/volume1/compose` | +| `/mnt/backup` | NFS — `10.100.50.50:/volume1/Backup` | + +## Running stacks + +| Stack | Port(s) | Notes | +|-------|---------|-------| +| adguard | 53 / 853 / 8080 / 8443 / 3000 / 784 | AdGuard Home — primary DNS for the NH site | +| dockge | 5001 | Docker stack management UI | +| dozzle-agent | 7007 | Log agent; hub on ana-docker pulls from here | +| beszel-nh3-docker (agent) | 45876 | Metrics agent; hub on ana-docker | +| portainer (+ portainer_agent) | 9443 / 8000 / 9001 | Container management UI (ad-hoc, no compose dir under `/opt/docker/compose/`) | + +No Traefik / TLS terminator on this host — services are accessed on raw LAN ports. If that changes, mirror the ana-docker pattern (`/opt/docker/conf/traefik-nh3/…` + `anaprod`-style cert resolver). + +## Refresh state + +```bash +scripts/refresh-server-info.sh nh3-docker +``` + +Latest snapshot: `system-details.txt` (regenerate as needed). + +## Placement rule + +No GPU → this is the default target for NH-site Docker workloads. GPU-only stacks still go to **ana-ml2**; non-GPU Anaheim-specific services stay on **ana-docker**. diff --git a/servers/nh3-docker/ssh-target b/servers/nh3-docker/ssh-target new file mode 100644 index 0000000..f2ddb45 --- /dev/null +++ b/servers/nh3-docker/ssh-target @@ -0,0 +1 @@ +10.100.50.40 diff --git a/servers/nh3-docker/system-details.txt b/servers/nh3-docker/system-details.txt new file mode 100644 index 0000000..1c5c123 --- /dev/null +++ b/servers/nh3-docker/system-details.txt @@ -0,0 +1,383 @@ + +===== HOST ===== + +Hostname: nh3-docker.phasefinal.com +Date: 2026-04-19T22:16:00-07:00 +Uptime: up 5 weeks, 4 days, 15 hours, 4 minutes +OS: Debian GNU/Linux 12 (bookworm) +Kernel: 6.1.0-21-amd64 +Arch: x86_64 + +===== HARDWARE ===== + +CPU cores: 8 +CPU model: QEMU Virtual CPU version 2.5+ +MemTotal: 7.8 GB +MemAvailable: 6.5 GB + +===== GPUS ===== + +nvidia-smi not present (no NVIDIA GPUs or driver not installed) + +===== FILESYSTEMS (df) ===== + +Filesystem Size Used Avail Use% Mounted on +/dev/sda1 125G 6.5G 112G 6% / + +===== PERSISTENT MOUNTS (/etc/fstab, non-comment) ===== + +UUID=dc0e74f7-3973-4501-b04e-1d9a3888f739 / ext4 errors=remount-ro 0 1 +UUID=cc35604a-fcb3-41e8-9a30-1b85e28eea99 none swap sw 0 0 +/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0 +10.100.50.50:/volume1/compose /mnt/compose nfs defaults 0 0 +10.100.50.50:/volume1/Backup /mnt/backup nfs defaults 0 0 + +===== TARGETED DATA PATHS ===== + +/opt (total: 1.6M) + total 12 + drwxr-xr-x 3 root root 4096 2024-05-30 00:24 . + drwxr-xr-x 18 root root 4096 2024-05-14 14:44 .. + drwxrwxrwx 5 root root 4096 2024-05-30 19:12 docker + +/opt/docker (total: 1.6M) + total 32 + drwxrwxrwx 5 root root 4096 2024-05-30 19:12 . + drwxr-xr-x 3 root root 4096 2024-05-30 00:24 .. + drwxr-xr-x 6 lkraven lkraven 4096 2026-04-19 01:03 compose + drwxr-xr-x 2 lkraven lkraven 4096 2024-05-30 19:11 conf + drwxr-xr-x 8 lkraven lkraven 4096 2024-05-31 19:05 .git + -rw-r--r-- 1 lkraven lkraven 14 2024-05-30 19:12 .gitignore + -rw-r--r-- 1 lkraven lkraven 1059 2024-05-30 00:27 LICENSE + -rw-r--r-- 1 lkraven lkraven 60 2024-05-30 00:27 README.md + +/opt/docker/compose (total: 56K) + total 28 + drwxr-xr-x 6 lkraven lkraven 4096 2026-04-19 01:03 . + drwxrwxrwx 5 root root 4096 2024-05-30 19:12 .. + drwxr-xr-x 2 lkraven lkraven 4096 2024-05-30 19:14 adguard + drwxr-xr-x 2 root root 4096 2026-04-19 01:03 beszel-nh3-docker + drwxr-xr-x 2 lkraven lkraven 4096 2024-05-30 19:14 dockge + drwxr-xr-x 2 root root 4096 2026-04-19 00:44 dozzle-agent + -rw-r--r-- 1 lkraven lkraven 25 2024-05-30 00:27 README.md + +/opt/docker/conf (total: 4.0K) + total 8 + drwxr-xr-x 2 lkraven lkraven 4096 2024-05-30 19:11 . + drwxrwxrwx 5 root root 4096 2024-05-30 19:12 .. + +/var/lib/docker (total: 4.0K) + +/srv (total: 4.0K) + total 8 + drwxr-xr-x 2 root root 4096 2024-05-14 14:43 . + drwxr-xr-x 18 root root 4096 2024-05-14 14:44 .. + + +===== DOCKER ===== + +Server: 20.10.24+dfsg1 Client: 20.10.24+dfsg1 + +----- docker info ----- +Containers: 6 (running 6, paused 0, stopped 0) +Images: 13 +Runtimes: map[io.containerd.runc.v2:{runc [] } io.containerd.runtime.v1.linux:{runc [] } runc:{runc [] }] +Default runtime: runc +Storage driver: overlay2 +Root dir: /var/lib/docker +Server version: 20.10.24+dfsg1 + +----- running containers ----- +NAMES IMAGE STATUS PORTS +beszel-agent henrygd/beszel-agent:latest Up 21 hours +dozzle-agent amir20/dozzle:latest Up 22 hours 0.0.0.0:7007->7007/tcp, 8080/tcp +dockge-dockge-1 louislam/dockge:latest Up 5 weeks (healthy) 0.0.0.0:5001->5001/tcp, :::5001->5001/tcp +adguardhome adguard/adguardhome Up 5 weeks 67-68/udp, 0.0.0.0:53->53/udp, :::53->53/udp, 443/udp, 0.0.0.0:53->53/tcp, :::53->53/tcp, 853/udp, 0.0.0.0:853->853/tcp, :::853->853/tcp, 3000/udp, 5443/tcp, 0.0.0.0:3000->3000/tcp, 0.0.0.0:784->784/udp, :::3000->3000/tcp, :::784->784/udp, 5443/udp, 6060/tcp, 0.0.0.0:8080->80/tcp, :::8080->80/tcp, 0.0.0.0:8443->443/tcp, :::8443->443/tcp +portainer_agent portainer/agent:2.19.4 Up 5 weeks 0.0.0.0:9001->9001/tcp, :::9001->9001/tcp +portainer portainer/portainer-ce:latest Up 5 weeks 0.0.0.0:8000->8000/tcp, :::8000->8000/tcp, 0.0.0.0:9443->9443/tcp, :::9443->9443/tcp, 9000/tcp + +----- all containers ----- +NAMES IMAGE STATUS +beszel-agent henrygd/beszel-agent:latest Up 21 hours +dozzle-agent amir20/dozzle:latest Up 22 hours +dockge-dockge-1 louislam/dockge:latest Up 5 weeks (healthy) +adguardhome adguard/adguardhome Up 5 weeks +portainer_agent portainer/agent:2.19.4 Up 5 weeks +portainer portainer/portainer-ce:latest Up 5 weeks + +----- networks ----- +NAME DRIVER SCOPE +bridge bridge local +host host local +none null local +traefik-net bridge local + +----- networks (external, non-default — worth knowing for compose external: true) ----- +traefik-net + +----- named volumes ----- +VOLUME NAME DRIVER +adguard_adguard-confdir local +adguard_adguard-workdir local +beszel-nh3-docker_beszel_agent_data local +dockge_dockge_data local +dozzle-agent_dozzle_agent_data local +portainer_data local + +----- compose projects currently running ----- +adguard +beszel-nh3-docker +dockge +dozzle-agent + +===== COMPOSE FILES (/opt/docker/compose/) ===== + + +>>> /opt/docker/compose/adguard/docker-compose.yml +#version: "3" +services: + adguardhome: + image: adguard/adguardhome + container_name: adguardhome + ports: + - 53:53/tcp + - 53:53/udp + - 784:784/udp + - 853:853/tcp + - 3000:3000/tcp + - 8080:80/tcp + - 8443:443/tcp + volumes: + - adguard-workdir:/opt/adguardhome/work + - adguard-confdir:/opt/adguardhome/conf + restart: unless-stopped + labels: + - homepage.group=PFI-NH3 + - homepage.name=AdGuard Home + - homepage.icon=si-adguard + - homepage.description=DNS + - homepage.href=http://10.100.50.40:8080 + - homepage.widget.type=adguard + - homepage.widget.url=http://10.100.50.40:8080 + - homepage.widget.username=lkraven + - homepage.widget.password=${userpass} + networks: + - tnet + env_file: + - .env +volumes: + adguard-workdir: null + adguard-confdir: null +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/beszel-nh3-docker/compose.yaml +# Beszel — lightweight server/container monitoring. +# +# Hub: single web UI with the SQLite store. Agents: per-host metric collectors +# that the hub pulls from over SSH. +# +# Multi-host layout via compose profiles: +# COMPOSE_PROFILES=hub → hub only (ana-docker) +# COMPOSE_PROFILES=hub,agent → hub + local agent on the same host +# COMPOSE_PROFILES=agent → agent only (ana-ml2) +# +# The agent uses network_mode: host so it sees real host CPU/mem/net/disk +# counters rather than container-scoped ones — that's why it can't share +# the tnet network with the hub. +# +# All tunables live in .env — edit that, not this file. + +services: + beszel: + image: henrygd/beszel:${BESZEL_VERSION} + container_name: beszel + profiles: [hub] + restart: unless-stopped + ports: + - "${BESZEL_PORT}:8090" + volumes: + - beszel_data:/beszel_data + healthcheck: + test: ["CMD", "wget", "-qO-", "http://localhost:8090/api/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 15s + networks: + - tnet + labels: + - homepage.group=PFI-ANA + - homepage.name=Beszel + - homepage.icon=mdi-chart-line + - homepage.description=Server + container monitoring + - homepage.href=http://10.250.50.70:${BESZEL_PORT} + + beszel-agent: + image: henrygd/beszel-agent:${BESZEL_VERSION} + container_name: beszel-agent + profiles: [agent] + restart: unless-stopped + network_mode: host + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - beszel_agent_data:/var/lib/beszel-agent + environment: + - PORT=${BESZEL_AGENT_PORT:-45876} + - KEY=${BESZEL_HUB_KEY} + - HUB_URL=${HUB_URL} + - TOKEN=${BESZEL_TOKEN} + - EXTRA_FILESYSTEMS=${BESZEL_EXTRA_FS:-} + +volumes: + beszel_data: + beszel_agent_data: + +networks: + tnet: + name: traefik-net + external: true +>>> /opt/docker/compose/dockge/compose.yaml +services: + dockge: + image: louislam/dockge:latest + restart: unless-stopped + ports: + # Host Port : Container Port + - 5001:5001 + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - dockge_data:/app/data + - /opt/docker/compose:/opt/docker/compose + environment: + # Tell Dockge where is your stacks directory + - DOCKGE_STACKS_DIR=/opt/docker/compose + labels: + - homepage.group=PFI-NH3 + - homepage.name=Dockge + - homepage.icon=si-portainer + - homepage.description=Docker + - homepage.href=http://10.100.50.40:5001 + networks: + - tnet +volumes: + dockge_data: null +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/dozzle-agent/compose.yaml +# Dozzle — container log viewer. +# +# Multi-host layout via compose profiles: +# COMPOSE_PROFILES=hub → runs the web UI (deploy on ana-docker) +# COMPOSE_PROFILES=agent → runs the remote agent (deploy on ana-ml2) +# +# Same compose.yaml on both servers; per-host `.env` picks the profile. +# +# All tunables live in .env — edit that, not this file. + +services: + dozzle: + image: amir20/dozzle:${DOZZLE_VERSION} + container_name: dozzle + profiles: + - hub + restart: unless-stopped + ports: + - ${DOZZLE_PORT}:8080 + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - dozzle_data:/data + environment: + - DOZZLE_HOSTNAME=${DOZZLE_HOSTNAME} + - DOZZLE_REMOTE_AGENT=${DOZZLE_REMOTE_AGENT:-} + - DOZZLE_AUTH_PROVIDER=${DOZZLE_AUTH_PROVIDER:-none} + - DOZZLE_USERNAME=${DOZZLE_USERNAME:-} + - DOZZLE_PASSWORD=${DOZZLE_PASSWORD:-} + healthcheck: + test: + - CMD + - /dozzle + - healthcheck + interval: 30s + timeout: 10s + retries: 3 + start_period: 15s + networks: + - tnet + labels: + - homepage.group=PFI-ANA + - homepage.name=Dozzle + - homepage.icon=mdi-text-box-search + - homepage.description=Container logs (ana-docker + ana-ml2) + - homepage.href=http://10.250.50.70:${DOZZLE_PORT} + dozzle-agent: + image: amir20/dozzle:${DOZZLE_VERSION} + container_name: dozzle-agent + profiles: + - agent + restart: unless-stopped + command: agent + ports: + - ${DOZZLE_AGENT_BIND:-0.0.0.0}:${DOZZLE_AGENT_PORT}:7007 + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - dozzle_agent_data:/data + environment: + - DOZZLE_HOSTNAME=${DOZZLE_HOSTNAME} + networks: + - tnet +volumes: + dozzle_data: null + dozzle_agent_data: null +networks: + tnet: + name: traefik-net + external: true +===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) ===== + +/opt/docker/conf + +===== LISTENING PORTS ===== + +0.0.0.0:111 +0.0.0.0:22 +0.0.0.0:3000 +0.0.0.0:5001 +0.0.0.0:53 +0.0.0.0:7007 +0.0.0.0:8000 +0.0.0.0:8080 +0.0.0.0:8443 +0.0.0.0:853 +0.0.0.0:9001 +0.0.0.0:9443 +[::]:111 +127.0.0.1:40185 +[::]:22 +*:2375 +[::]:3000 +[::]:5001 +[::]:53 +[::]:8000 +[::]:8080 +[::]:8443 +[::]:853 +[::]:9001 +[::]:9443 + +===== MODEL / HUGGINGFACE CACHES ===== + + +===== DOCKER-ADJACENT SYSTEMD SERVICES ===== + +containerd.service running +docker.service running + +===== DONE ===== + +Paste the above back into the chat, or pass a path as argv[1] to save. diff --git a/stacks/backrest/.env.example b/stacks/backrest/.env.example new file mode 100644 index 0000000..486b595 --- /dev/null +++ b/stacks/backrest/.env.example @@ -0,0 +1,17 @@ +# backrest stack tunables. Copy to `.env` on ana-docker before deploying. +# +# cp .env.example .env +# # edit if needed +# docker compose up -d +# +# Repos, S3 endpoints, restic passwords, and schedules are all configured +# inside the Backrest web UI after first boot — nothing belongs here. + +# Image version — pin for reproducibility (`latest` for edge) +BACKREST_VERSION=latest + +# Host port for the web UI (container listens on 9898 internally) +BACKREST_PORT=9898 + +# Timezone — affects scheduler display and log timestamps +TZ=America/Los_Angeles diff --git a/stacks/backrest/README.md b/stacks/backrest/README.md new file mode 100644 index 0000000..9e48385 --- /dev/null +++ b/stacks/backrest/README.md @@ -0,0 +1,65 @@ +# backrest + +Web UI over restic repositories. Runs **once, on ana-docker**, and points at every per-host restic repo on both site-local S3 endpoints to give a single pane of glass for snapshot history, restores, and alerts. + +**Server:** ana-docker +**Port:** `http://10.250.50.70:9898` + +## Role in the fleet + +- Actual backups are run by per-host `systemd` timers calling `restic` (see the backup design; not yet deployed). Each host writes to its site-local S3 bucket (TrueNAS on ana, Synology on nh3). +- **This container does not run backups by default** — it's a viewer/manager pointed at existing repos. (Backrest *can* be the scheduler instead of systemd timers; we're keeping the scheduler on the host for simplicity.) +- Because it only needs to talk to S3 endpoints (not host filesystems), no bind mounts of host paths are required. + +## Deploy + +```bash +# On ana-docker: +sudo mkdir -p /opt/docker/compose/backrest +sudo chown $USER /opt/docker/compose/backrest +cd /opt/docker/compose/backrest + +# scp compose.yaml + .env.example from this workspace, then: +cp .env.example .env +# edit .env if you want a different port or TZ + +docker compose config +docker compose up -d +docker compose logs -f +``` + +Open `http://10.250.50.70:9898` and set the admin credentials on first load. + +## First-time configuration (in the UI) + +For each per-host repo (to be added once the restic pipeline is running): + +1. **Add Repository** → fill in: + - **ID:** e.g. `ana-docker`, `ana-ml2`, `nh3-docker`, `esh-docker-vm` + - **URI:** `s3:https://10.250.50.50:4521/pfi-backups/ana/ana-docker/` (or the nh3 endpoint, depending on host) + - **Password:** the restic passphrase for that host + - **Env:** `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` for the S3 endpoint +2. *(Do not create a plan unless you want Backrest to drive the schedule — leave schedules to the systemd timers on each host.)* +3. Verify: the repo should list snapshots pulled by the host's systemd timer within a few minutes. + +## Backup scope, when wired up + +- ana-docker, ana-ml2 → TrueNAS S3 at `10.250.50.50:4521` +- nh3-docker, esh-docker-vm → Synology S3 at `10.100.50.50:4521` +- Cross-site rclone sync makes each bucket also hold the other site's data, so you can restore *either* host from *either* side if one NAS is down. + +## Scaling knobs + +- **Upgrade:** `docker compose pull && docker compose up -d`. +- **Backup of Backrest itself:** its config/DB lives in the `backrest_config` and `backrest_data` volumes — include those in ana-docker's restic plan so recreating the UI doesn't mean re-entering every repo. + +## Why not `offen/docker-volume-backup`? + +Two instances on `esh-docker-vm` (paperless-ngx, pgadmin) currently use the `offen/docker-volume-backup` sidecar pattern. Those will be retired once restic is in place: + +- No encryption — backups sit in the clear on NFS. +- No dedup — every run writes a full tarball; storage grows linearly. +- Per-stack config — every new service needs its own sidecar wiring. +- No cross-host index — restores require knowing which tarball lives where. + +Restic + Backrest solves all four at the cost of one extra binary per host. Leave the sidecars running until the restic plan is verified, then remove them in a scheduled change. diff --git a/stacks/backrest/compose.yaml b/stacks/backrest/compose.yaml new file mode 100644 index 0000000..518a694 --- /dev/null +++ b/stacks/backrest/compose.yaml @@ -0,0 +1,58 @@ +# Backrest — web UI over restic repositories. +# +# Role here: single central viewer for every host's restic repo on both +# site-local S3 endpoints (TrueNAS at ana, Synology at nh3). Per-host +# `restic` runs will still be driven by systemd timers on each host; this +# stack is how we see what ran, browse snapshots, and restore. +# +# Repos and S3 credentials are configured in the Backrest UI after first +# boot — nothing baked into this file. Data (its own SQLite + queue) lives +# in a named volume so the config survives container recreation. +# +# All tunables live in .env — edit that, not this file. + +services: + backrest: + image: garethgeorge/backrest:${BACKREST_VERSION} + container_name: backrest + hostname: backrest + restart: unless-stopped + ports: + - "${BACKREST_PORT}:9898" + volumes: + - backrest_data:/data + - backrest_config:/config + - backrest_cache:/cache + - backrest_tmp:/tmp + environment: + - BACKREST_DATA=/data + - BACKREST_CONFIG=/config/config.json + - XDG_CACHE_HOME=/cache + - TMPDIR=/tmp + - TZ=${TZ:-America/Los_Angeles} + - BACKREST_PORT=0.0.0.0:9898 + healthcheck: + test: ["CMD", "wget", "-qO-", "http://localhost:9898/"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 15s + networks: + - tnet + labels: + - homepage.group=PFI-ANA + - homepage.name=Backrest + - homepage.icon=mdi-backup-restore + - homepage.description=Restic snapshot viewer / restore UI + - homepage.href=http://10.250.50.70:${BACKREST_PORT} + +volumes: + backrest_data: + backrest_config: + backrest_cache: + backrest_tmp: + +networks: + tnet: + name: traefik-net + external: true diff --git a/stacks/beszel/.env.example b/stacks/beszel/.env.example new file mode 100644 index 0000000..d7e4b86 --- /dev/null +++ b/stacks/beszel/.env.example @@ -0,0 +1,43 @@ +# beszel stack tunables. Copy to `.env` on each server before deploying. +# +# cp .env.example .env +# # edit for this host +# docker compose up -d +# +# Same compose.yaml on both servers — COMPOSE_PROFILES picks the role. + +# Image version — pin for reproducibility (`latest` for edge) +BESZEL_VERSION=latest + +# ------------------------------------------------------------------------ +# On ana-docker (hub + local agent): +# COMPOSE_PROFILES=hub,agent +# BESZEL_PORT=8090 +# BESZEL_HUB_KEY= +# +# On ana-ml2 (agent only): +# COMPOSE_PROFILES=agent +# BESZEL_HUB_KEY= +# ------------------------------------------------------------------------ + +COMPOSE_PROFILES=hub,agent + +# ---- Hub-only ----------------------------------------------------------- + +# Host port for the web UI (container listens on 8090 internally). +BESZEL_PORT=8090 + +# ---- Agent-only --------------------------------------------------------- + +# Host port the agent listens on. The hub SSHes into agents over this port. +BESZEL_AGENT_PORT=45876 + +# Hub's SSH public key — paste from the hub UI on first run. +# Grab it by clicking "Add System" → copy the key shown in the dialog. +BESZEL_HUB_KEY= + +# Extra filesystems to track beyond the root mount, comma-separated. +# Examples: +# on ana-ml2: /tank +# on ana-docker: /mnt/backup,/mnt/compose +BESZEL_EXTRA_FS= diff --git a/stacks/beszel/README.md b/stacks/beszel/README.md new file mode 100644 index 0000000..cae0f98 --- /dev/null +++ b/stacks/beszel/README.md @@ -0,0 +1,88 @@ +# beszel + +Lightweight monitoring — CPU, memory, disk, network, and per-container stats for every Docker host, with alerts over email/webhook. Pairs with Dozzle (logs) on the same server. + +**Deploys to:** +- **ana-docker** (hub + local agent) — UI at `http://10.250.50.70:8090` +- **ana-ml2** (agent only) — listens on `10.250.50.54:45876` +- **nh3-docker** (agent only, cross-site) — listens on `10.100.50.40:45876` + +Same compose.yaml on each host. Per-host `.env` sets `COMPOSE_PROFILES` to bring up the right combination. Each agent host is added individually in the hub UI. + +## How hub ↔ agent auth works + +Beszel uses SSH-key-based auth: the hub generates its own keypair on first boot, and each agent must be seeded with the hub's **public key** via the `KEY` env var. Agents listen on a port (default 45876); the hub pulls metrics by connecting to them with that key. + +Operator flow on first deploy: + +1. Bring up the **hub** on ana-docker with `BESZEL_HUB_KEY=` blank and the agent profile disabled. +2. Open the UI, create the admin account, click **Add System** — Beszel shows the public key. +3. Copy the key into `BESZEL_HUB_KEY` in the `.env` on both hosts. +4. Re-deploy the hub with `COMPOSE_PROFILES=hub,agent` to add the local agent; deploy the agent on ana-ml2. +5. Back in the UI, **Add System** with `host=127.0.0.1 port=45876` (local) and `host=10.250.50.54 port=45876` (ana-ml2). + +## Deploy — hub + local agent (ana-docker) + +```bash +ssh ana-docker +sudo mkdir -p /opt/docker/compose/beszel +sudo chown $USER /opt/docker/compose/beszel +cd /opt/docker/compose/beszel + +# scp compose.yaml + .env.example, then: +cp .env.example .env +# First pass — hub only, no key yet: +# COMPOSE_PROFILES=hub +# BESZEL_PORT=8090 +docker compose up -d + +# Open http://10.250.50.70:8090 → create admin → click "Add System" → +# copy the displayed public key into BESZEL_HUB_KEY. + +# Second pass — add the local agent: +# COMPOSE_PROFILES=hub,agent +# BESZEL_EXTRA_FS=/mnt/backup,/mnt/compose +docker compose up -d +``` + +## Deploy — agent (ana-ml2) + +```bash +ssh ana-ml2 +sudo mkdir -p /opt/docker/compose/beszel +sudo chown $USER /opt/docker/compose/beszel +cd /opt/docker/compose/beszel + +# scp the same compose.yaml + .env.example, then: +cp .env.example .env +# Edit to: +# COMPOSE_PROFILES=agent +# BESZEL_HUB_KEY= +# BESZEL_EXTRA_FS=/tank + +docker compose up -d +docker compose logs -f +``` + +Then in the hub UI, **Add System** with `host=10.250.50.54`, `port=45876`. + +## Verify + +```bash +# Hub health +curl -s http://10.250.50.70:8090/api/health + +# Agent reachable +ssh ana-docker 'nc -zv 10.250.50.54 45876' + +# Local agent reachable from hub container +docker exec beszel nc -zv host.docker.internal 45876 +``` + +## Sizing / impact + +The agent is ~10 MB RAM and negligible CPU — runs fine alongside anything on ana-ml2 including GPU workloads. Host-mode networking means it has no port conflicts with other stacks as long as `BESZEL_AGENT_PORT` stays unique. + +## Alerts + +Configured inside the hub UI (Settings → Notifications). Supports email (SMTP), Gotify, ntfy, Discord, Slack, and generic webhooks. Alert rules attach to per-system or global thresholds (CPU, memory, disk, container down, etc.). diff --git a/stacks/beszel/compose.yaml b/stacks/beszel/compose.yaml new file mode 100644 index 0000000..dafadf3 --- /dev/null +++ b/stacks/beszel/compose.yaml @@ -0,0 +1,63 @@ +# Beszel — lightweight server/container monitoring. +# +# Hub: single web UI with the SQLite store. Agents: per-host metric collectors +# that the hub pulls from over SSH. +# +# Multi-host layout via compose profiles: +# COMPOSE_PROFILES=hub → hub only (ana-docker) +# COMPOSE_PROFILES=hub,agent → hub + local agent on the same host +# COMPOSE_PROFILES=agent → agent only (ana-ml2) +# +# The agent uses network_mode: host so it sees real host CPU/mem/net/disk +# counters rather than container-scoped ones — that's why it can't share +# the tnet network with the hub. +# +# All tunables live in .env — edit that, not this file. + +services: + beszel: + image: henrygd/beszel:${BESZEL_VERSION} + container_name: beszel + profiles: [hub] + restart: unless-stopped + ports: + - "${BESZEL_PORT}:8090" + volumes: + - beszel_data:/beszel_data + healthcheck: + test: ["CMD", "wget", "-qO-", "http://localhost:8090/api/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 15s + networks: + - tnet + labels: + - homepage.group=PFI-ANA + - homepage.name=Beszel + - homepage.icon=mdi-chart-line + - homepage.description=Server + container monitoring + - homepage.href=http://10.250.50.70:${BESZEL_PORT} + + beszel-agent: + image: henrygd/beszel-agent:${BESZEL_VERSION} + container_name: beszel-agent + profiles: [agent] + restart: unless-stopped + network_mode: host + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - beszel_agent_data:/var/lib/beszel-agent + environment: + - PORT=${BESZEL_AGENT_PORT:-45876} + - KEY=${BESZEL_HUB_KEY} + - EXTRA_FILESYSTEMS=${BESZEL_EXTRA_FS:-} + +volumes: + beszel_data: + beszel_agent_data: + +networks: + tnet: + name: traefik-net + external: true diff --git a/stacks/dozzle/.env.example b/stacks/dozzle/.env.example new file mode 100644 index 0000000..bc9debe --- /dev/null +++ b/stacks/dozzle/.env.example @@ -0,0 +1,50 @@ +# dozzle stack tunables. Copy to `.env` on each server before deploying. +# +# cp .env.example .env +# # edit for this host +# docker compose up -d +# +# Same compose.yaml on both servers — COMPOSE_PROFILES picks the role. + +# Image version — pin for reproducibility (`latest` for edge) +DOZZLE_VERSION=latest + +# ------------------------------------------------------------------------ +# On ana-docker (hub): +# COMPOSE_PROFILES=hub +# DOZZLE_HOSTNAME=ana-docker +# DOZZLE_REMOTE_AGENT=10.250.50.54:7007 +# +# On ana-ml2 (agent): +# COMPOSE_PROFILES=agent +# DOZZLE_HOSTNAME=ana-ml2 +# ------------------------------------------------------------------------ + +COMPOSE_PROFILES=hub + +# Display name for this host in the UI (shown as a tab / section header). +DOZZLE_HOSTNAME=ana-docker + +# ---- Hub-only ----------------------------------------------------------- + +# Host port for the web UI (container listens on 8080 internally). +DOZZLE_PORT=8088 + +# Comma-separated list of remote agents the hub should connect to. +# Leave blank if this host only views its own containers. +DOZZLE_REMOTE_AGENT=10.250.50.54:7007,10.100.50.40:7007 + +# Auth — `none` is fine behind the LAN / a reverse proxy with auth. +# Switch to `simple` and set USERNAME/PASSWORD to gate the UI itself. +DOZZLE_AUTH_PROVIDER=none +DOZZLE_USERNAME= +DOZZLE_PASSWORD= + +# ---- Agent-only --------------------------------------------------------- + +# Host port the agent listens on (container listens on 7007 internally). +DOZZLE_AGENT_PORT=7007 + +# Bind address — restrict to the LAN interface if you want belt-and-braces +# beyond what the firewall already enforces. Default 0.0.0.0 exposes on all. +DOZZLE_AGENT_BIND=0.0.0.0 diff --git a/stacks/dozzle/README.md b/stacks/dozzle/README.md new file mode 100644 index 0000000..5e00391 --- /dev/null +++ b/stacks/dozzle/README.md @@ -0,0 +1,74 @@ +# dozzle + +Container log viewer. One UI on **ana-docker** aggregates logs from every Docker host via remote agents. + +**Deploys to:** +- **ana-docker** (hub) — UI at `http://10.250.50.70:8088` +- **ana-ml2** (agent) — listens on `10.250.50.54:7007` +- **nh3-docker** (agent, cross-site) — listens on `10.100.50.40:7007` + +One compose.yaml lives on each host. The per-host `.env` sets `COMPOSE_PROFILES=hub` or `COMPOSE_PROFILES=agent` so `docker compose up -d` brings up the right service. On the hub, add every agent to `DOZZLE_REMOTE_AGENT` as a comma-separated list (e.g. `10.250.50.54:7007,10.100.50.40:7007`). + +## Auth / TLS note + +Dozzle agents and hub auto-generate mTLS certificates on first run. On the trusted LAN (10.250.0.0/16) the default config is fine. If you ever expose an agent beyond the LAN, generate and pin certificates explicitly per the Dozzle docs (`dozzle generate`). The web UI itself is unauthenticated by default — flip `DOZZLE_AUTH_PROVIDER=simple` and set `DOZZLE_USERNAME`/`DOZZLE_PASSWORD` in the hub `.env` if you want a login gate. + +## Deploy — hub (ana-docker) + +```bash +ssh ana-docker +sudo mkdir -p /opt/docker/compose/dozzle +sudo chown $USER /opt/docker/compose/dozzle +cd /opt/docker/compose/dozzle + +# scp compose.yaml + .env.example from this workspace, then: +cp .env.example .env +# Ensure: +# COMPOSE_PROFILES=hub +# DOZZLE_HOSTNAME=ana-docker +# DOZZLE_REMOTE_AGENT=10.250.50.54:7007 +# DOZZLE_PORT=8088 + +docker compose config +docker compose up -d +docker compose logs -f +``` + +## Deploy — agent (ana-ml2) + +```bash +ssh ana-ml2 +sudo mkdir -p /opt/docker/compose/dozzle +sudo chown $USER /opt/docker/compose/dozzle +cd /opt/docker/compose/dozzle + +# scp the same compose.yaml + .env.example, then: +cp .env.example .env +# Edit to: +# COMPOSE_PROFILES=agent +# DOZZLE_HOSTNAME=ana-ml2 +# DOZZLE_AGENT_PORT=7007 + +docker compose config +docker compose up -d +docker compose logs -f +``` + +## Verify + +```bash +# Hub health (from anywhere on LAN) +curl -s http://10.250.50.70:8088/healthz + +# Agent reachable from the hub's perspective +ssh ana-docker 'nc -zv 10.250.50.54 7007' + +# Open http://10.250.50.70:8088 — you should see two tabs: +# "ana-docker" (local containers) and "ana-ml2" (via agent). +``` + +## Troubleshooting + +- **Hub shows only local containers:** agent is unreachable. Check firewall rules on ana-ml2 (port 7007 must be open from 10.250.50.70) and that the agent is actually listening (`ss -tlnp | grep 7007`). +- **Agent keeps restarting:** verify the docker.sock bind mount is read-only and the socket exists. +- **Certificate mismatch after image upgrade:** delete the `dozzle_data` and `dozzle_agent_data` volumes on both hosts and redeploy to regenerate. diff --git a/stacks/dozzle/compose.yaml b/stacks/dozzle/compose.yaml new file mode 100644 index 0000000..59f82d3 --- /dev/null +++ b/stacks/dozzle/compose.yaml @@ -0,0 +1,66 @@ +# Dozzle — container log viewer. +# +# Multi-host layout via compose profiles: +# COMPOSE_PROFILES=hub → runs the web UI (deploy on ana-docker) +# COMPOSE_PROFILES=agent → runs the remote agent (deploy on ana-ml2) +# +# Same compose.yaml on both servers; per-host `.env` picks the profile. +# +# All tunables live in .env — edit that, not this file. + +services: + dozzle: + image: amir20/dozzle:${DOZZLE_VERSION} + container_name: dozzle + profiles: [hub] + restart: unless-stopped + ports: + - "${DOZZLE_PORT}:8080" + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - dozzle_data:/data + environment: + - DOZZLE_HOSTNAME=${DOZZLE_HOSTNAME} + - DOZZLE_REMOTE_AGENT=${DOZZLE_REMOTE_AGENT:-} + - DOZZLE_AUTH_PROVIDER=${DOZZLE_AUTH_PROVIDER:-none} + - DOZZLE_USERNAME=${DOZZLE_USERNAME:-} + - DOZZLE_PASSWORD=${DOZZLE_PASSWORD:-} + healthcheck: + test: ["CMD", "/dozzle", "healthcheck"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 15s + networks: + - tnet + labels: + - homepage.group=PFI-ANA + - homepage.name=Dozzle + - homepage.icon=mdi-text-box-search + - homepage.description=Container logs (ana-docker + ana-ml2) + - homepage.href=http://10.250.50.70:${DOZZLE_PORT} + + dozzle-agent: + image: amir20/dozzle:${DOZZLE_VERSION} + container_name: dozzle-agent + profiles: [agent] + restart: unless-stopped + command: agent + ports: + - "${DOZZLE_AGENT_BIND:-0.0.0.0}:${DOZZLE_AGENT_PORT}:7007" + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - dozzle_agent_data:/data + environment: + - DOZZLE_HOSTNAME=${DOZZLE_HOSTNAME} + networks: + - tnet + +volumes: + dozzle_data: + dozzle_agent_data: + +networks: + tnet: + name: traefik-net + external: true diff --git a/stacks/infinity/.env.example b/stacks/infinity/.env.example new file mode 100644 index 0000000..47c4157 --- /dev/null +++ b/stacks/infinity/.env.example @@ -0,0 +1,30 @@ +# Infinity stack tunables. Copy this to `.env` on the server before deploying. +# +# cp .env.example .env +# # edit .env with real values +# docker compose up -d + +# Image version — pin for reproducibility (`latest` for edge) +INFINITY_VERSION=latest + +# Port exposed on host +INFINITY_PORT=7997 + +# GPU assignment (ana-ml2 has 0 and 1; default 1 keeps 0 free for heavy LLM work) +GPU_ID=1 + +# Models — both served simultaneously; reference by the full repo name in requests +EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B +RERANK_MODEL=Qwen/Qwen3-Reranker-0.6B + +# Inference engine: torch (widest support) or optimum (ONNX, sometimes faster) +ENGINE=torch + +# Batch size — 32 is a safe default; bump for throughput if VRAM allows +BATCH_SIZE=32 + +# Optional API key — leave blank for no auth (fine on the internal network) +API_KEY= + +# HuggingFace token — only needed for gated models +HF_TOKEN= diff --git a/stacks/infinity/README.md b/stacks/infinity/README.md new file mode 100644 index 0000000..c4e3db8 --- /dev/null +++ b/stacks/infinity/README.md @@ -0,0 +1,65 @@ +# infinity + +OpenAI-compatible embeddings + reranker server. One container serves both embedding and reranker models simultaneously. + +**Server:** ana-ml2 +**Port:** 7997 (infinity default) +**GPU:** pinned to GPU 1 by default (configurable via `.env`) + +## What it replaces / supersedes + +- `qwen3-embedding-0.6B` entry in llama-swap (llama.cpp GGUF → infinity transformer) +- `qwen3-reranker-0.6B` entry in llama-swap + +Once infinity is verified stable, retire those two entries from `stacks/llama-swap/config.yaml`. + +## Deploy + +```bash +# On ana-ml2: +sudo mkdir -p /opt/docker/compose/infinity +sudo chown $USER /opt/docker/compose/infinity +cd /opt/docker/compose/infinity + +# Copy compose.yaml + .env.example here (e.g. via scp from this workspace) +# Then: +cp .env.example .env +# edit .env — pick GPU, models, etc. + +# Pre-download models into the shared HF cache (optional, speeds first boot) +HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^EMBED_MODEL .env | cut -d= -f2)" +HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^RERANK_MODEL .env | cut -d= -f2)" + +# Dry-parse +docker compose config + +# Launch +docker compose up -d +docker compose logs -f +``` + +## Verify + +```bash +# Health +curl -s http://localhost:7997/health + +# Embedding +curl -s http://localhost:7997/embeddings \ + -H "Content-Type: application/json" \ + -d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":["hello world"]}' | jq . + +# Reranker +curl -s http://localhost:7997/rerank \ + -H "Content-Type: application/json" \ + -d '{"model":"Qwen/Qwen3-Reranker-0.6B","query":"what is a cat","documents":["cats are mammals","dogs bark"]}' | jq . + +# Listed models +curl -s http://localhost:7997/models | jq . +``` + +## Scaling knobs + +- **`BATCH_SIZE`** in `.env` — bigger = higher throughput, more VRAM. 32 is safe; try 64 or 128 if you have headroom. +- **Model size** — Qwen3-Embedding/Reranker come in 0.6B / 4B / 8B. Pick based on quality-vs-latency tradeoff. On RTX 6000 Ada 46 GB, the 8B pair fits easily (~20 GB VRAM). +- **`ENGINE=optimum`** — uses ONNX runtime, sometimes faster. Requires the model to have ONNX weights available; fall back to `torch` if it errors on startup. diff --git a/stacks/infinity/compose.yaml b/stacks/infinity/compose.yaml new file mode 100644 index 0000000..16e799f --- /dev/null +++ b/stacks/infinity/compose.yaml @@ -0,0 +1,63 @@ +# Infinity — OpenAI-compatible embeddings + reranker server. +# +# Serves embedding and reranker models simultaneously from one container +# on port 7997 (HTTP). Consumers: AIPA agents (search/retrieval), LibreChat +# RAG, anything that needs vector embeddings. +# +# All tunables live in .env — edit that, not this file. +# +# Pre-download models to avoid first-run delay: +# HF_HOME=/tank/aimodels/huggingface hf download Qwen/Qwen3-Embedding-0.6B +# HF_HOME=/tank/aimodels/huggingface hf download Qwen/Qwen3-Reranker-0.6B + +services: + infinity: + image: michaelf34/infinity:${INFINITY_VERSION} + container_name: infinity + restart: unless-stopped + ports: + - "${INFINITY_PORT}:7997" + volumes: + - /tank/aimodels/huggingface:/hfcache + environment: + - HF_HOME=/hfcache + - HF_HUB_CACHE=/hfcache/hub + - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-} + command: > + v2 + --model-id ${EMBED_MODEL} + --model-id ${RERANK_MODEL} + --engine ${ENGINE} + --device cuda + --batch-size ${BATCH_SIZE} + --host 0.0.0.0 + --port 7997 + --api-key ${API_KEY:-} + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: + - "${GPU_ID}" + capabilities: + - gpu + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:7997/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 120s + networks: + - tnet + labels: + - homepage.group=AI Systems + - homepage.name=Infinity + - homepage.icon=mdi-vector-arrange-below + - homepage.description=Embeddings + Reranker API (ana-ml2) + - homepage.href=http://10.250.50.54:7997/docs + +networks: + tnet: + name: traefik-net + external: true diff --git a/stacks/llama-swap/.env.example b/stacks/llama-swap/.env.example new file mode 100644 index 0000000..09ce372 --- /dev/null +++ b/stacks/llama-swap/.env.example @@ -0,0 +1,18 @@ +# llama-swap stack tunables. Copy to `.env` on ana-docker before deploying. +# +# cp .env.example .env +# # edit if needed +# docker compose up -d + +# Image tag. `cuda` is the CUDA-enabled build; pin to a specific release +# (e.g. `cuda-v0.0.6`) for reproducibility once upstream tags stabilize. +LLAMA_SWAP_VERSION=cuda + +# Host port for the OpenAI-compatible API (container listens on 8080) +LLAMA_SWAP_PORT=9292 + +# Host paths +# --- Legacy GGUF models referenced by config.yaml as `-m /models/` +MODELS_DIR=/tank/aimodels/llm +# --- Shared HuggingFace cache used by `-hf` model entries +HF_CACHE_DIR=/tank/aimodels/huggingface diff --git a/stacks/llama-swap/README.md b/stacks/llama-swap/README.md new file mode 100644 index 0000000..c04a7e6 --- /dev/null +++ b/stacks/llama-swap/README.md @@ -0,0 +1,52 @@ +# llama-swap + +GGUF model server with on-demand model swapping. Served via llama.cpp's `llama-server` under the llama-swap proxy. + +**Server:** ana-ml2 +**Port:** 9292 (configurable via `.env`) +**GPU:** both (unpinned — `runtime: nvidia` grants access to all devices; per-model GPU selection happens inside `config.yaml`) + +## Files + +- **`compose.yaml`** — canonical compose. Deployed to `/opt/docker/compose/llama-swap/compose.yaml` on ana-ml2. +- **`.env.example`** — template for the per-host `.env`. Copy to `.env` on the server and tweak. +- **`config.yaml`** — model definitions and groups. Deployed to `/opt/docker/conf/llama-swap/config.yaml` on the server. + +Homepage labels are in the compose file under the `AI Systems` group, matching the convention used by `vllm-qwen3` and `infinity`. + +## Deploy a fresh install + +```bash +scripts/deploy-stack.sh ana-ml2 llama-swap + +ssh ana-ml2 ' + cd /opt/docker/compose/llama-swap && \ + cp -n .env.example .env && \ + docker compose config && \ + docker compose up -d && \ + docker compose logs --tail=30 +' +``` + +## Model reference conventions + +- **Modern entries:** use `-hf /[:]` — reads from the shared HF cache, nothing to pre-stage outside `hf download` +- **Legacy entries:** use `--model /models//.gguf` — reads GGUFs from `/tank/aimodels/llm/` (pre-HF-cache era, gradually being migrated) + +New models should prefer the `-hf` pattern. + +## Deploy updates to config only + +```bash +# After editing config.yaml here: +scp config.yaml ana-ml2:/opt/docker/conf/llama-swap/config.yaml +ssh ana-ml2 'cd /opt/docker/compose/llama-swap && docker compose restart' +``` + +## Deploy updates to compose only + +```bash +# After editing compose.yaml or .env.example here: +scripts/deploy-stack.sh ana-ml2 llama-swap +ssh ana-ml2 'cd /opt/docker/compose/llama-swap && docker compose up -d' +``` diff --git a/stacks/llama-swap/compose.yaml b/stacks/llama-swap/compose.yaml new file mode 100644 index 0000000..c1c547c --- /dev/null +++ b/stacks/llama-swap/compose.yaml @@ -0,0 +1,50 @@ +# llama-swap — GGUF model server with on-demand model swapping. +# +# Proxies OpenAI-compatible API requests to llama.cpp server instances +# and swaps which model is loaded into VRAM per request. Runs on +# ana-ml2 using both GPUs dynamically (no explicit device pinning — +# llama-swap picks per-model-definition). +# +# Model definitions live in /opt/docker/conf/llama-swap/config.yaml on +# the server. Canonical copy of that config is config.yaml in this +# workspace; deploy with scp + `docker compose restart` or the script +# at the bottom of README.md. +# +# All tunables live in .env — edit that, not this file. + +services: + llama-swap: + image: ghcr.io/mostlygeek/llama-swap:${LLAMA_SWAP_VERSION} + container_name: llama-swap + restart: unless-stopped + stdin_open: true + tty: true + runtime: nvidia + ports: + - "${LLAMA_SWAP_PORT}:8080" + volumes: + - /opt/docker/conf/llama-swap/config.yaml:/app/config.yaml + - ${MODELS_DIR}:/models + - ${HF_CACHE_DIR}:/hfcache + environment: + - HF_HOME=/hfcache + - HF_HUB_CACHE=/hfcache/hub + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:8080/ >/dev/null || exit 1"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 30s + networks: + - tnet + labels: + - homepage.group=AI Systems + - homepage.name=llama-swap + - homepage.icon=mdi-swap-horizontal + - homepage.description=GGUF model swapper (llama.cpp; ana-ml2) + - homepage.href=http://10.250.50.54:${LLAMA_SWAP_PORT} + +networks: + tnet: + name: traefik-net + external: true diff --git a/stacks/llama-swap/config.yaml b/stacks/llama-swap/config.yaml new file mode 100644 index 0000000..2484738 --- /dev/null +++ b/stacks/llama-swap/config.yaml @@ -0,0 +1,463 @@ +# ============================================================================ +# llama-swap configuration for PFI-ANA +# Optimized and synchronized with /models disk inventory +# Last updated: 2026-04-10 +# +# KB Sources: +# - reference/nemotron-3-super-running-parameters.md +# - reference/nemotron-3-nano-running-parameters.md +# - reference/qwen3.5-running-parameters.md +# - reference/qwen3-coder-next-running-parameters.md +# - reference/gemma-4-running-parameters.md +# - reference/qwen3-embedding-running-parameters.md +# - reference/qwen3-reranker-running-parameters.md +# +# Changelog: +# 2025-07-22: Removed jina-reranker-v3 (unused, out of rotation). +# 2026-04-10: Fixed Qwen3-Embedding pooling (mean→last; causal LM uses last-token +# pooling). Fixed ctx-size 4096→8192 for embedding+reranker. Fixed +# reranker: removed --embeddings flag (not an embedding model). +# 2026-04-17: Added Qwen3.6-35B-A3B Abliterated Heretic Q8_0 via -hf syntax. +# Requires HF_HOME=/hfcache in compose (see docker-compose.yml). +# New convention: use -hf repo[:quant] instead of --model /path. +# ============================================================================ + +# Default 1200 seconds (20 min) to wait for model to be available to load. +healthCheckTimeout: 1200 + +# logLevel: sets the logging value +# - optional, default: info +# - Valid log levels: debug, info, warn, error +logLevel: info + +# metricsMaxInMemory: maximum number of metrics to keep in memory +# - optional, default: 1000 +metricsMaxInMemory: 1000 + +# startPort: sets the starting port number for the automatic ${PORT} macro. +# - optional, default: 5800 +# - the ${PORT} macro can be used in model.cmd and model.proxy settings +# - it is automatically incremented for every model that uses it +# startPort: 10001 + +models: + + # ========================================================================== + # QWEN 3.5 MODELS (KB-recommended settings) + # - Thinking mode: temp 1.0, top-p 0.95, top-k 20, min-p 0.0, presence_penalty 1.5 + # - Coding (precise): temp 0.6, top-p 0.95, top-k 20, min-p 0.0, presence_penalty 0.0 + # - Non-thinking general: temp 0.7, top-p 0.8, top-k 20, min-p 0.0, presence_penalty 1.5 + # - Context: 256K native (start 16K-32K for responsiveness) + # - Gibberish fix: add --cache-type-k bf16 --cache-type-v bf16 + # - No Ollama support for Qwen3.5 GGUFs — use llama.cpp only + # ========================================================================== + + "qwen3.5-35-a3b": + name: "Qwen 3.5 35B-A3B Thinking" + description: "MoE reasoning model. 3B active params, general-purpose thinking/chat." + ttl: 600 + cmd: | + /app/llama-server + --context-shift + --model /models/unsloth_Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf + --port ${PORT} + --n-gpu-layers 999 + --ctx-size 32768 + --flash-attn on + --temp 1.0 + --top-p 0.95 + --top-k 20 + --min-p 0.00 + --presence-penalty 1.5 + --chat-template-kwargs '{"enable_thinking":true}' + + "qwen3.5-122b-a10b": + name: "Qwen 3.5 122B-A10B UD-Q4_K_XL" + description: "Large MoE reasoning model. 10B active params, heavy reasoning tasks." + ttl: 600 + cmd: | + /app/llama-server + --context-shift + --model /models/unsloth_Qwen3.5-122B-A10B-GGUF/UD-Q4_K_XL/Qwen3.5-122B-A10B-UD-Q4_K_XL-00001-of-00003.gguf + --port ${PORT} + --n-gpu-layers 999 + --ctx-size 32768 + --flash-attn on + --temp 1.0 + --top-p 0.95 + --top-k 20 + --min-p 0.00 + --presence-penalty 1.5 + --chat-template-kwargs '{"enable_thinking":true}' + + "qwen3.5-9b": + name: "Qwen 3.5 9B UD-Q4_K_XL" + description: "Dense 9B model. Lightweight general-purpose chat and reasoning." + ttl: 600 + cmd: | + /app/llama-server + --context-shift + --model /models/unsloth_Qwen3.5-9B-GGUF/Qwen3.5-9B-UD-Q4_K_XL.gguf + --port ${PORT} + --n-gpu-layers 999 + --ctx-size 32768 + --flash-attn on + --temp 1.0 + --top-p 0.95 + --top-k 20 + --min-p 0.00 + --presence-penalty 1.5 + --chat-template-kwargs '{"enable_thinking":true}' + + # -------------------------------------------------------------------------- + # Qwen 3.6 — uses -hf syntax, reads from HF_HOME=/hfcache (host pre-download) + # -------------------------------------------------------------------------- + + "qwen3.6-35-a3b-abliterated": + name: "Qwen 3.6 35B-A3B Abliterated Heretic Q8_0" + description: "Qwen3.6 MoE, 3B active. Abliterated/heretic variant of BF16 quantized to Q8_0. ~38GB." + ttl: 600 + cmd: | + /app/llama-server + --context-shift + --jinja + -hf IIEleven11/Qwen3.6-35B-A3B-Abliterated-Heretic-BF16-Q8_0-GGUF + --port ${PORT} + --n-gpu-layers 999 + --ctx-size 32768 + --flash-attn on + --temp 1.0 + --top-p 0.95 + --top-k 20 + --min-p 0.00 + --presence-penalty 1.5 + --repeat-penalty 1.0 + --reasoning on + --reasoning-format deepseek + + # ========================================================================== + # NEMOTRON MODELS (KB-recommended settings) + # - General Chat: temp 1.0, top-p 1.0, min_p 0.01 + # - Tool Calling: temp 0.6, top-p 0.95, min_p 0.01 + # - NoPE architecture: no YaRN needed + # - DEPRECATED --special flag for reasoning tokens + # - --special flag causes issues. + # - Start ctx 16K-32K, increase cautiously + # ========================================================================== + + "nemotron-3-super-120b": + name: "NVIDIA Nemotron 3 Super 120B-A12B UD-Q4_K_XL" + description: "Flagship NVIDIA reasoning model. 12B active of 120B, MoE. 64-72GB VRAM at Q4." + ttl: 600 + cmd: | + /app/llama-server + --context-shift + --model /models/unsloth_NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_XL/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-00001-of-00003.gguf + --port ${PORT} + --n-gpu-layers 999 + --ctx-size 16384 + --flash-attn on + --temp 1.0 + --top-p 1.0 + --min-p 0.01 + --seed 3407 + + "nemotron-3-nano-30b": + name: "NVIDIA Nemotron 3 Nano 30B-A3B UD-Q4_K_XL" + description: "Compact Nemotron. 3B active of 30B, MoE. ~24GB at Q4. Best performance/size on 24GB GPUs." + ttl: 600 + cmd: | + /app/llama-server + --context-shift + --special + --model /models/unsloth_Nemotron-3-Nano-30B-A3B-GGUF/Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf + --port ${PORT} + --n-gpu-layers 999 + --ctx-size 32768 + --flash-attn on + --temp 1.0 + --top-p 1.0 + --min-p 0.01 + --seed 3407 + + # ========================================================================== + # GEMMA 4 MODELS (KB-recommended settings) + # - All variants: temp 1.0, top-p 0.95, top-k 64, repeat_penalty 1.0 + # - Thinking: enable via --chat-template-kwargs '{"enable_thinking":true}' + # - Multi-turn: only keep final visible answer in history (not thought blocks) + # - Context: E2B/E4B=128K, 26B-A4B/31B=256K. Start at 32K. + # - ⚠️ Do NOT use CUDA 13.2 runtime — causes poor outputs + # - Use llama-server (not llama-cli) for thinking control + # ========================================================================== + + "gemma4-26b-a4b": + name: "Gemma 4 26B-A4B" + description: "Google DeepMind Gemma 4 MoE, 4B active params, 256K context. Best speed/quality tradeoff." + ttl: 600 + cmd: | + /app/llama-server + --context-shift + --model /models/unsloth_gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf + --port ${PORT} + --n-gpu-layers 999 + --ctx-size 32768 + --flash-attn on + --temp 1.0 + --top-p 0.95 + --top-k 64 + --repeat-penalty 1.0 + --chat-template-kwargs '{"enable_thinking":true}' + + "gemma4-31b-dense": + name: "Gemma 4 31B Dense" + description: "Google DeepMind Gemma 4 dense 31B. Maximum quality for complex reasoning, 256K context." + ttl: 600 + cmd: | + /app/llama-server + --context-shift + --model /models/unsloth_gemma-4-31B-it-GGUF/gemma-4-31B-it-UD-Q4_K_XL.gguf + --port ${PORT} + --n-gpu-layers 999 + --ctx-size 32768 + --flash-attn on + --temp 1.0 + --top-p 0.95 + --top-k 64 + --repeat-penalty 1.0 + --chat-template-kwargs '{"enable_thinking":true}' + + # ========================================================================== + # GLM MODELS + # ========================================================================== + + "glm4.7-flash": + name: "GLM 4.7 Flash UD-Q4_K_XL" + description: "THUDM GLM 4.7 Flash. Fast inference, general-purpose chat." + ttl: 600 + cmd: | + /app/llama-server + --context-shift + --model /models/unsloth_GLM-4.7-Flash-GGUF/GLM-4.7-Flash-UD-Q4_K_XL.gguf + --port ${PORT} + --n-gpu-layers 999 + --ctx-size 40000 + --flash-attn on + --temp 0.6 + --top-p 0.95 + + "glm-steam-106b": + name: "GLM Steam 106B-A12B Q4_K_M" + description: "TheDrummer GLM Steam MoE. 12B active of 106B. Creative and RP-focused." + ttl: 600 + cmd: | + /app/llama-server + --context-shift + --model /models/RP/bartowski_TheDrummer_GLM-Steam-106B-A12B-v1-GGUF/TheDrummer_GLM-Steam-106B-A12B-v1-Q4_K_M-00001-of-00002.gguf + --port ${PORT} + --n-gpu-layers 999 + --ctx-size 40000 + --flash-attn on + --temp 0.6 + --top-p 0.95 + + # ========================================================================== + # SKYFALL MODELS + # ========================================================================== + + "skyfall-r1-31b-q6k": + name: "Skyfall R1 31B v4 Q6_K_L" + description: "TheDrummer Skyfall R1 31B v4. General-purpose reasoning." + ttl: 600 + cmd: | + /app/llama-server + --context-shift + --model /models/bartowski_TheDrummer_Skyfall-31B-v4-GGUF/TheDrummer_Skyfall-31B-v4-Q6_K_L.gguf + --port ${PORT} + --n-gpu-layers 999 + --ctx-size 40000 + --flash-attn on + + "skyfall-r1-31b-v4a": + name: "Skyfall R1 31B v4a Q6_K (RP)" + description: "BeaverAI Skyfall R1 v4a variant. RP/creative-focused." + ttl: 600 + cmd: | + /app/llama-server + --context-shift + --model /models/RP/BeaverAI_Skyfall-R1-31B-v4a-GGUF/Skyfall-R1-31B-v4a-Q6_K.gguf + --port ${PORT} + --n-gpu-layers 999 + --ctx-size 40000 + --flash-attn on + + # ========================================================================== + # CODER MODELS + # ========================================================================== + + "qwen3-coder-next": + name: "Qwen3 Coder Next UD-Q4_K_XL" + description: "Latest Qwen3 Coder. Non-reasoning model, optimized for code gen. KB: temp 1.0, top-k 40, min-p 0.01." + ttl: 600 + cmd: | + /app/llama-server + --context-shift + --model /models/unsloth_Qwen3-Coder-Next-GGUF/Qwen3-Coder-Next-UD-Q4_K_XL.gguf + --port ${PORT} + --n-gpu-layers 999 + --ctx-size 32768 + --flash-attn on + --temp 1.0 + --top-p 0.95 + --top-k 40 + --min-p 0.01 + --repeat-penalty 1.0 + + # ========================================================================== + # LARGE / SPECIAL-PURPOSE MODELS + # ========================================================================== + + "kimik2-q2kxl": + name: "Kimi K2 Instruct UD-Q2_K_XL" + description: "Moonshot Kimi K2. Huge MoE model (8-shard Q2). Limited GPU layers due to size." + ttl: 600 + cmd: | + /app/llama-server + --context-shift + --model /models/unsloth_Kimi-K2-Instruct-0905-GGUF/UD-Q2_K_XL/Kimi-K2-Instruct-0905-UD-Q2_K_XL-00001-of-00008.gguf + --port ${PORT} + --n-gpu-layers 2 + --temp 0.6 + --top-p 0.95 + + # ========================================================================== + # GRANITE MODELS (IBM) + # ========================================================================== + + "granite-4-small": + name: "Granite 4.0 Small Q4_K_M" + description: "IBM Granite 4.0 Small. Deterministic utility model for structured tasks." + ttl: 0 + cmd: | + /app/llama-server + --context-shift + --model /models/unsloth_granite-4.0-h-small-GGUF/granite-4.0-h-small-Q4_K_M.gguf + --port ${PORT} + --n-gpu-layers 999 + --ctx-size 120000 + --flash-attn on + --top-p 1.0 + --temp 0.0 + --top-k 0 + + "granite-4-micro": + name: "Granite 4.0 Micro Q4_K_M" + description: "IBM Granite 4.0 Micro. Ultra-lightweight for fast structured responses." + ttl: 600 + cmd: | + /app/llama-server + --context-shift + --model /models/ibm-granite_granite-4.0-micro-GGUF/granite-4.0-micro-Q4_K_M.gguf + --port ${PORT} + --n-gpu-layers 999 + --ctx-size 32768 + --flash-attn on + --temp 0.0 + --top-p 1.0 + + # ========================================================================== + # EMBEDDING MODELS (persistent, always loaded) + # ========================================================================== + + "embeddinggemma-300M": + name: "Embedding Gemma 300M" + description: "Google Embedding Gemma for vectorization." + ttl: 0 + cmd: | + /app/llama-server + --embedding + --pooling cls + --model /models/ggml-org_embeddinggemma-300M-GGUF/embeddinggemma-300M-Q8_0.gguf + --port ${PORT} + --n-gpu-layers 0 + --ctx-size 2048 + --batch-size 1024 + --no-mmap + --ubatch-size 1024 + --cont-batching + --threads 24 + + "qwen3-embedding-0.6B": + name: "Qwen3 Embedding 0.6B" + description: "Qwen3 Embedding model for vectorization. 32K context, last-token pooling (decoder/causal LM)." + ttl: 0 + cmd: | + /app/llama-server + --embeddings + --pooling last + --model /models/Qwen_Qwen3-Embedding-0.6B-GGUF/Qwen3-Embedding-0.6B-Q8_0.gguf + --port ${PORT} + --n-gpu-layers 0 + --ctx-size 8192 + --batch-size 8192 + --ubatch-size 2048 + --no-mmap + --cont-batching + --threads 24 + + # ========================================================================== + # RERANKING MODELS (persistent, always loaded) + # ========================================================================== + + "qwen3-reranker-0.6B": + name: "Qwen3 Reranker 0.6B" + description: "Qwen3 Reranker for retrieval reranking. Causal LM scoring yes/no logits at last token. Replaces BGE v2." + ttl: 0 + cmd: | + /app/llama-server + --reranking + --pooling rank + --model /models/ggml-org_Qwen3-Reranker-0.6B-Q8_0-GGUF/qwen3-reranker-0.6b-q8_0.gguf + --port ${PORT} + --n-gpu-layers 0 + --ctx-size 8192 + --batch-size 8192 + --ubatch-size 2048 + --cont-batching + --threads 24 + + # NOTE: jina-reranker-v3 removed 2025-07-22 — unused, out of rotation. + # Qwen3 Reranker handles all reranking duties. + +# ============================================================================ +# GROUPS +# - swap: false = models in group can coexist in memory +# - exclusive: false = group can share memory with other groups +# - persistent: true = models never unload (for utility/embedding) +# ============================================================================ + +groups: + "high-reasoning": + swap: false + exclusive: false + members: + - "qwen3.5-35-a3b" + - "gemma4-31b-dense" + - "nemotron-3-nano-30b" + + "heavy-moe": + swap: false + exclusive: false + members: + - "qwen3.5-122b-a10b" + - "nemotron-3-super-120b" + - "kimik2-q2kxl" + - "glm-steam-106b" + + "utility": + swap: false + exclusive: false + persistent: true + members: + - "embeddinggemma-300M" + - "qwen3-embedding-0.6B" + - "qwen3-reranker-0.6B" diff --git a/stacks/rest-server-ana/.env.example b/stacks/rest-server-ana/.env.example new file mode 100644 index 0000000..19feafd --- /dev/null +++ b/stacks/rest-server-ana/.env.example @@ -0,0 +1,41 @@ +# rest-server-ana stack tunables. Copy to `.env` on ana-docker. +# +# cp .env.example .env +# # edit if needed +# docker compose up -d +# +# Matches stacks/rest-server-nh3/.env.example — keep them aligned so +# restic clients see the same URL shape against either endpoint. + +REST_SERVER_VERSION=latest + +# Host port the container listens on (container internal is 8000) +REST_PORT=8000 + +# Where restic pack files live. TrueNAS NFS share is already mounted at +# /mnt/backup on ana-docker; the "repo/ana" subdir is the historical +# location used by the prior (non-private-repos) rest-server. +# +# With --private-repos, layout becomes: +# ${DATA_DIR}/ana-docker/ +# ${DATA_DIR}/ana-ml2/ +# ${DATA_DIR}/nh3-docker/ +# ${DATA_DIR}/esh-docker-vm/ +# Plus the auth file at ${DATA_DIR}/.htpasswd. +DATA_DIR=/mnt/backup/restic/repo/ana + +# UID/GID the container process runs as. Must match the owner of +# DATA_DIR so NFS root_squash doesn't bite. On ana-docker this is the +# `lkraven` user (1000:1000). +REST_UID=1000 +REST_GID=1000 + +# Timezone — affects log lines and /metrics timestamps +TZ=America/Los_Angeles + +# Extra rest-server flags. Examples: +# --prometheus-no-auth — make /metrics public (needed if Beszel or +# Prometheus scrapes without creds) +# --no-verify-upload — trust the client's hash; faster writes +# Leave blank unless you have a reason. +EXTRA_OPTIONS= diff --git a/stacks/rest-server-ana/README.md b/stacks/rest-server-ana/README.md new file mode 100644 index 0000000..42d76b3 --- /dev/null +++ b/stacks/rest-server-ana/README.md @@ -0,0 +1,143 @@ +# rest-server-ana + +Anaheim-site restic backup endpoint. Replaces the older `restic` stack on ana-docker with the same auth model as `rest-server-nh3` on the Synology, so every client host uses identical URL shapes against either endpoint. + +**Server:** ana-docker (`10.250.50.70`) +**Port:** `http://10.250.50.70:8000` +**Data:** `/mnt/backup/restic/repo/ana/` (TrueNAS NFS mount on the host) + +Paired with: +- **`rest-server-nh3`** on the Synology (`10.100.50.50:8000`, data on Btrfs). +- A cross-site rsync job (TBD, on ana-docker) that mirrors each site's data tree to the other so either NAS can fully restore either site's hosts. + +## What changed from the old `restic` stack + +| | old `restic` on ana-docker | this stack | +|---|---|---| +| `--private-repos` | no | **yes** | +| `--append-only` | no | **yes** | +| `--prometheus` | no | **yes** | +| healthcheck | no | yes | +| `.env`-driven | no | yes | +| restart policy | none | `unless-stopped` | +| image version | floating `latest` | `${REST_SERVER_VERSION}` | +| stack dir on server | `/opt/docker/compose/restic/` | `/opt/docker/compose/rest-server-ana/` | + +Data path is unchanged (`/mnt/backup/restic/repo/ana/`) so nothing new needs to be allocated on TrueNAS. + +## Pre-deploy: clean the data dir and create htpasswd + +Since there's nothing in the existing path we want to keep, start fresh so the on-disk layout matches `--private-repos`: + +```bash +ssh ana-docker ' + # Stop the old stack so port 8000 and the data dir are free + cd /opt/docker/compose/restic + docker compose down + + # Wipe the old non-private-repos layout + sudo rm -rf /mnt/backup/restic/repo/ana/* + sudo rm -rf /mnt/backup/restic/repo/ana/.htpasswd # if present + + # Create the htpasswd file. Use the same passwords here as on the NH3 + # Synology so each host has one credential that works at either endpoint. + sudo touch /mnt/backup/restic/repo/ana/.htpasswd + sudo chmod 600 /mnt/backup/restic/repo/ana/.htpasswd +' + +# Generate htpasswd entries locally (one per host) and append. Using the +# `httpd:2.4-alpine` throwaway container so we do not depend on +# apache2-utils being installed on ana-docker. +for user in ana-docker ana-ml2 nh3-docker esh-docker-vm; do + read -rs -p "password for $user (must match the NH3 Synology): " pw; echo + docker run --rm httpd:2.4-alpine htpasswd -nbB "$user" "$pw" \ + | ssh ana-docker 'sudo tee -a /mnt/backup/restic/repo/ana/.htpasswd >/dev/null' +done +``` + +If you run that locally and don't have Docker here, equivalent on the server: + +```bash +ssh ana-docker "docker run --rm httpd:2.4-alpine htpasswd -nbB ''" \ + | ssh ana-docker 'sudo tee -a /mnt/backup/restic/repo/ana/.htpasswd >/dev/null' +``` + +## Deploy + +Stage the new stack and push it: + +```bash +# Stage the stack into the mirror (if not already done via sync-stacks.sh) +mkdir -p stacks-mirror/ana-docker/rest-server-ana +cp stacks/rest-server-ana/compose.yaml stacks/rest-server-ana/.env.example \ + stacks-mirror/ana-docker/rest-server-ana/ + +scripts/deploy-stack.sh ana-docker rest-server-ana +``` + +Confirm at the prompt. Then on the server: + +```bash +ssh ana-docker ' + cd /opt/docker/compose/rest-server-ana + cp -n .env.example .env + docker compose config + docker compose up -d + docker compose logs --tail=30 +' +``` + +## Retire the old stack + +Once the new one is healthy and the first repo has initialized successfully from a client: + +```bash +ssh ana-docker ' + cd /opt/docker/compose/restic + docker compose down + # Optionally remove the old stack dir (keep it for a release or two + # in case you need to roll back): + # rm -rf /opt/docker/compose/restic +' +``` + +## Verify + +```bash +# 401 from the root — service up, auth enforced +curl -sS -o /dev/null -w 'unauth status=%{http_code}\n' \ + http://10.250.50.70:8000/ + +# 200 / 404 from a real user+password — auth valid, --private-repos path OK +curl -sS -o /dev/null -w 'auth status=%{http_code}\n' \ + -u ana-docker: http://10.250.50.70:8000/ana-docker/ + +# Init a repo from a client host (one-time per host) +ssh ana-docker ' + export RESTIC_REPOSITORY="rest:http://ana-docker:@10.250.50.70:8000/ana-docker/" + export RESTIC_PASSWORD="" + restic init +' +``` + +## Prune ceremony + +Same as `rest-server-nh3` — prune is blocked by `--append-only`. Two options, pick one per endpoint: + +- **Temporary flag flip:** edit compose, remove `--append-only` from `OPTIONS`, `docker compose up -d`, run `restic forget --prune` from origin hosts, put the flag back, `docker compose up -d`. Quarterly change. +- **Second endpoint on a different port:** stand up a sibling container (e.g. port `8001`) against the same data dir without `--append-only`, reachable only from a trusted host. Everyday backups still hit `:8000`. + +If you go the second-endpoint route, copy this stack to `stacks/rest-server-ana-prune/` with `REST_PORT=8001` and `--append-only` removed from the compose. + +## Off-site replication + +Scheduled on ana-docker (to be written): + +```bash +# Pull NH3's tree down to this side +rsync -avz --delete admin@10.100.50.50:/volume1/Backup/restic/ /mnt/backup/restic-mirror-nh3/ +# Push our tree to NH3 +rsync -avz --delete /mnt/backup/restic/repo/ana/ admin@10.100.50.50:/volume1/Backup/restic-mirror-ana/ +``` + +Two unidirectional syncs, each running in the direction its data flows. Prune runs only at the origin so the mirror shrinks correctly. diff --git a/stacks/rest-server-ana/compose.yaml b/stacks/rest-server-ana/compose.yaml new file mode 100644 index 0000000..cdfac3e --- /dev/null +++ b/stacks/rest-server-ana/compose.yaml @@ -0,0 +1,59 @@ +# rest-server (Anaheim) — restic backup target for the fleet. +# +# Deploys to ana-docker. Data dir is on the TrueNAS NFS mount +# (/mnt/backup/restic/repo/ana) so snapshots on the NAS side protect the +# backup blobs themselves. +# +# Mirrors stacks/rest-server-nh3/ in every meaningful way — same auth +# model, same on-disk layout, same operational semantics — so each client +# host uses an identical URL shape against either endpoint: +# +# rest:http://:@10.100.50.50:8000// (NH3 Synology) +# rest:http://:@10.250.50.70:8000// (this stack) +# +# Auth model: +# --private-repos : URL path must start with // and the HTTP +# basic-auth user must match. Per-host repos are +# strictly isolated. +# --append-only : on-disk data can be added but not removed or +# rewritten; a compromised host can't wipe its own +# history. Prune requires disabling this (see README). +# +# Credentials come from /data/.htpasswd — see README for populating it. +# +# All tunables live in .env — edit that, not this file. + +services: + rest-server: + image: restic/rest-server:${REST_SERVER_VERSION} + container_name: rest-server + restart: unless-stopped + # Run as the UID that owns the NFS-backed data dir, so file I/O + # is not subject to NFS root_squash. On ana-docker this is lkraven (1000). + user: "${REST_UID:-1000}:${REST_GID:-1000}" + ports: + - "${REST_PORT}:8000" + volumes: + - ${DATA_DIR}:/data + environment: + - OPTIONS=--private-repos --append-only --prometheus ${EXTRA_OPTIONS:-} + - TZ=${TZ:-America/Los_Angeles} + healthcheck: + test: ["CMD-SHELL", "wget -qO- http://localhost:8000/metrics >/dev/null 2>&1 || [ $? -eq 6 ] && exit 0 || exit 1"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 15s + networks: + - tnet + labels: + - homepage.group=PFI-ANA + - homepage.name=Restic (rest-server) + - homepage.icon=mdi-cloud-upload + - homepage.description=Anaheim restic endpoint (data on TrueNAS NFS) + - homepage.href=http://10.250.50.70:${REST_PORT} + +networks: + tnet: + name: traefik-net + external: true diff --git a/stacks/rest-server-nh3/.env.example b/stacks/rest-server-nh3/.env.example new file mode 100644 index 0000000..cb32997 --- /dev/null +++ b/stacks/rest-server-nh3/.env.example @@ -0,0 +1,35 @@ +# rest-server (NH3 Synology) tunables. Copy to `.env` on the Synology. +# +# cp .env.example .env +# # edit if needed +# docker compose up -d +# +# Synology-specific notes: +# - Put this stack under /volume1/docker/compose/rest-server/ +# (Container Manager's default project root pattern on DSM 7.x). +# - Point DATA_DIR at a Btrfs share you own — /volume1/Backup/restic +# is the natural choice since /volume1/Backup is already the +# fleet-facing backup share (exported via NFS). + +REST_SERVER_VERSION=latest + +# Host port the container listens on (container internal port is 8000) +REST_PORT=8000 + +# Where restic pack files live on the Synology. Must be a writable +# Btrfs path. Will hold one subdir per user (--private-repos layout): +# ${DATA_DIR}/ana-docker/ +# ${DATA_DIR}/ana-ml2/ +# ${DATA_DIR}/nh3-docker/ +# ${DATA_DIR}/esh-docker-vm/ +# Plus the auth file at ${DATA_DIR}/.htpasswd. +DATA_DIR=/volume1/Backup/restic + +# Timezone — affects log lines and the /metrics timestamps +TZ=America/Los_Angeles + +# Any extra rest-server flags (rare). Some useful ones: +# --no-verify-upload — trust the client's hash; faster writes +# --max-size= — cap per-repo size +# Leave blank unless you have a reason. +EXTRA_OPTIONS= diff --git a/stacks/rest-server-nh3/README.md b/stacks/rest-server-nh3/README.md new file mode 100644 index 0000000..bd8440c --- /dev/null +++ b/stacks/rest-server-nh3/README.md @@ -0,0 +1,124 @@ +# rest-server-nh3 + +NH3-site restic backup endpoint. Runs in Synology Container Manager on `10.100.50.50` and stores pack files on a Btrfs share so Synology snapshots protect against local corruption. + +**Server:** Synology RS2418+ at `10.100.50.50` +**Port:** `http://10.100.50.50:8000` (configurable via `.env`) +**Data:** `/volume1/Backup/restic/` (configurable) + +Paired with the existing `restic rest-server` on **ana-docker** (`http://10.250.50.70:8000`, data on TrueNAS NFS) as the Anaheim-side endpoint. Each fleet host backs up to the rest-server closest to it; an rsync job on ana-docker mirrors the two trees against each other for off-site redundancy. + +## Auth model + +`--private-repos` + `--append-only`, enforced via htpasswd: + +- One HTTP basic-auth user **per host** (`ana-docker`, `ana-ml2`, `nh3-docker`, `esh-docker-vm`). +- Each user can only write under `//…` — a compromised host can't see or delete another host's data. +- Append-only means a compromised client can add to its own repo but can't rewrite or delete existing packs, so ransomware on a backed-up host doesn't destroy history. +- **Trade-off:** `restic forget --prune` can't run against an append-only endpoint. Prune ceremony described at the bottom of this file. + +## Pre-deploy: create the data path and htpasswd + +On the Synology (SSH in as an admin-capable user, or DSM *File Station*): + +```bash +# 1. Create the restic data share on a Btrfs volume +ssh admin@10.100.50.50 'sudo mkdir -p /volume1/Backup/restic && \ + sudo chown 1000:1000 /volume1/Backup/restic && \ + sudo chmod 700 /volume1/Backup/restic' + +# 2. Generate htpasswd entries. The Synology doesn't ship apache2-utils, +# so use a throwaway container: +ssh admin@10.100.50.50 'cd /volume1/Backup/restic && \ + sudo touch .htpasswd && sudo chown 1000:1000 .htpasswd && sudo chmod 600 .htpasswd' + +for user in ana-docker ana-ml2 nh3-docker esh-docker-vm; do + read -s -p "password for $user: " pw; echo + ssh admin@10.100.50.50 \ + "docker run --rm httpd:2.4-alpine htpasswd -nbB $user '$pw'" \ + | ssh admin@10.100.50.50 "sudo tee -a /volume1/Backup/restic/.htpasswd >/dev/null" +done +``` + +Record every password in your off-host password manager (1Password / Vaultwarden etc.) — you'll paste them into Backrest and into the systemd timer configs later. + +## Deploy + +In Synology **Container Manager**: + +1. *Project* → **Create** → Name `rest-server`, Path `/volume1/docker/compose/rest-server/`. +2. Copy `compose.yaml` into the project path; copy `.env.example` → `.env` and edit if needed (default `REST_PORT=8000` and `DATA_DIR=/volume1/Backup/restic` should be fine). +3. Start the project. + +CLI equivalent (if you have SSH + a shell account that can run Docker on the NAS): + +```bash +ssh admin@10.100.50.50 +sudo mkdir -p /volume1/docker/compose/rest-server +sudo chown $USER /volume1/docker/compose/rest-server +cd /volume1/docker/compose/rest-server + +# scp the files from this workspace, then: +cp .env.example .env +docker compose config +docker compose up -d +docker compose logs -f +``` + +## Verify + +From this workstation: + +```bash +# Should return "200 OK" or redirect to /metrics; anything 5xx is a problem. +curl -u ana-docker: -sv http://10.100.50.50:8000/ana-docker/ -o /dev/null + +# Once restic is wired up, init the repo (one-time, per host): +RESTIC_REPOSITORY='rest:http://ana-docker:@10.100.50.50:8000/ana-docker/' \ +RESTIC_PASSWORD='' \ + restic init +``` + +Restic URI shape for each host (paste into Backrest when adding the repo): + +``` +rest:http://:@10.100.50.50:8000// +``` + +## Prune ceremony (because of --append-only) + +Because `--append-only` blocks deletes, `restic forget --prune` will fail against the live endpoint. Two options, pick one and stick with it: + +### Option A — temporary flag flip (simplest, requires a maintenance window) + +1. On the Synology, edit the stack's `.env` and set `EXTRA_OPTIONS=--no-auth` — **only kidding, don't.** Set `EXTRA_OPTIONS= ` and comment out `--append-only` in the compose `OPTIONS=` line (or parameterize if you prefer). +2. `docker compose up -d` to restart with deletes allowed. +3. Run `restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-yearly 3 --prune` from the origin host. +4. Restore `--append-only` and `docker compose up -d`. + +Treat this as a quarterly change, not a cron job. Schedule it so you're present if restic hits anything weird. + +### Option B — second endpoint on a different port (automation-friendly) + +Stand up a second `rest-server` container against the same `DATA_DIR` without `--append-only`, listening on e.g. `8001`, reachable only from within the Synology / over VPN. A scheduled prune job hits that endpoint; day-to-day backup traffic continues to hit `:8000` in append-only mode. + +If you end up wanting this, copy this stack to `stacks/rest-server-nh3-prune/` with `REST_PORT=8001` and `--append-only` removed. + +## Off-site replication + +Scheduled on ana-docker: + +```bash +# Example — not the final script, just illustrating the shape. +rsync -avz --delete \ + admin@10.100.50.50:/volume1/Backup/restic/ \ + /mnt/backup/restic-mirror-nh3/ +``` + +`rsync` is safe because restic packs are immutable once written — nothing under `/volume1/Backup/restic//data/` gets rewritten, only added or (during prune) removed. A raw `rsync --delete` with prune running only on the origin side is enough; no filesystem-level locks required. + +## What doesn't live here + +- No `.env` in the committed copy — only `.env.example`. +- `.htpasswd` is never checked in, never synced via `sync-stacks.sh` (its `*.ht*` isn't in the global exclude but the data dir is outside the stack path). +- Client-side restic passwords (the encryption passphrase for each repo) are separate from the HTTP auth passwords and never stored on the Synology. diff --git a/stacks/rest-server-nh3/compose.yaml b/stacks/rest-server-nh3/compose.yaml new file mode 100644 index 0000000..43a89f9 --- /dev/null +++ b/stacks/rest-server-nh3/compose.yaml @@ -0,0 +1,40 @@ +# rest-server (NH3 Synology) — restic backup target for the fleet. +# +# Deploys into Synology Container Manager on 10.100.50.50. Data lives on +# a Btrfs shared folder so it gets Synology snapshots + optional +# replication to a sibling share if you configure one later. +# +# Auth model: +# --private-repos : every URL path must start with // and the +# HTTP basic-auth user must match. One user per host. +# Per-host repos are strictly isolated. +# --append-only : on-disk data can be ADDED but not REMOVED or REWRITTEN. +# A compromised host can't delete its own history. +# Prune requires disabling this (see README). +# +# Credentials come from /data/.htpasswd — see README for how to populate +# it. That file is mounted read-only into the container. +# +# All tunables live in .env — edit that, not this file. + +services: + rest-server: + image: restic/rest-server:${REST_SERVER_VERSION} + container_name: rest-server + restart: unless-stopped + ports: + - "${REST_PORT}:8000" + volumes: + - ${DATA_DIR}:/data + environment: + - OPTIONS=--private-repos --append-only --prometheus ${EXTRA_OPTIONS:-} + - TZ=${TZ:-America/Los_Angeles} + # rest-server stores repos under /data and looks for /data/.htpasswd + # automatically — no extra bind mount needed as long as the htpasswd + # file is created inside DATA_DIR before startup. + healthcheck: + test: ["CMD-SHELL", "wget -qO- http://localhost:8000/metrics >/dev/null || exit 1"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 15s diff --git a/stacks/vllm-qwen3/.env.example b/stacks/vllm-qwen3/.env.example new file mode 100644 index 0000000..2a655c7 --- /dev/null +++ b/stacks/vllm-qwen3/.env.example @@ -0,0 +1,40 @@ +# vllm-qwen3 stack tunables. Copy this to `.env` on the server before deploying. +# +# cp .env.example .env +# # edit .env with real values +# docker compose up -d + +# Image version — pin for reproducibility (`latest` for edge) +VLLM_VERSION=latest + +# Host ports (container always listens on 8000 internally) +EMBED_PORT=8001 +RERANK_PORT=8002 + +# GPU assignment — both services share this GPU +# (ana-ml2 has 0 and 1; default 1 keeps 0 free for heavy LLM work) +GPU_ID=1 + +# Models — reference by full repo name in API requests +EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B +RERANK_MODEL=Qwen/Qwen3-Reranker-0.6B + +# GPU memory split — fractions are of TOTAL GPU memory, not free memory. +# When two vLLM services share a GPU, each profiler needs its own slice to +# fit both the model and KV cache, so small values cause the second-to-start +# service to OOM on KV cache allocation. 0.40 + 0.40 leaves ~20% headroom +# and is comfortably above the minimum for two 0.6B Qwen3 models at 8k ctx. +EMBED_GPU_MEM_UTIL=0.40 +RERANK_GPU_MEM_UTIL=0.40 + +# Context length caps — lower these if VRAM is tight. +# Qwen3-Embedding supports up to 32k; reranker up to 32k. +EMBED_MAX_MODEL_LEN=8192 +RERANK_MAX_MODEL_LEN=8192 + +# Optional API key — leave blank for no auth (fine on the internal network). +# If set, both services require `Authorization: Bearer `. +API_KEY= + +# HuggingFace token — only needed for gated models +HF_TOKEN= diff --git a/stacks/vllm-qwen3/README.md b/stacks/vllm-qwen3/README.md new file mode 100644 index 0000000..f53fb42 --- /dev/null +++ b/stacks/vllm-qwen3/README.md @@ -0,0 +1,80 @@ +# vllm-qwen3 + +Qwen3 embedding + reranker served via vLLM. Replaces the unmaintained Infinity stack. + +**Server:** ana-ml2 +**Ports:** `8001` (embed), `8002` (rerank) — both configurable via `.env` +**GPU:** both services share GPU 1 by default (configurable) + +## Why two services + +vLLM runs **one model per process**, so embedding and reranking each get their own container. Both pin to the same GPU and split VRAM via `--gpu-memory-utilization`. Both use `--runner pooling` so the OpenAI server exposes `/v1/embeddings` (for the embedder) and `/rerank`, `/score` (for the reranker, which also needs the `--hf-overrides` described below). + +## Reranker caveat + +Qwen/Qwen3-Reranker-0.6B is a causal-LM checkpoint. The `--hf-overrides` flag in `compose.yaml` re-maps it to `Qwen3ForSequenceClassification` so vLLM's `/rerank` and `/score` endpoints work and the model emits only `no`/`yes` class logits instead of the full 151k-token distribution. + +If that override breaks after a vLLM upgrade, the pre-converted checkpoint `tomaarsen/Qwen3-Reranker-0.6B-seq-cls` is a drop-in replacement that needs no overrides — set `RERANK_MODEL=tomaarsen/Qwen3-Reranker-0.6B-seq-cls` in `.env` and remove the `--hf-overrides` line from the compose. + +## Deploy + +```bash +# On ana-ml2: +sudo mkdir -p /opt/docker/compose/vllm-qwen3 +sudo chown $USER /opt/docker/compose/vllm-qwen3 +cd /opt/docker/compose/vllm-qwen3 + +# Copy compose.yaml + .env.example here (e.g. via scp from this workspace) +cp .env.example .env +# edit .env — pick GPU, ports, memory split, etc. + +# Pre-download models (optional, speeds first boot) +HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^EMBED_MODEL .env | cut -d= -f2)" +HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^RERANK_MODEL .env | cut -d= -f2)" + +# Dry-parse +docker compose config + +# Launch +docker compose up -d +docker compose logs -f +``` + +First boot compiles CUDA graphs and can take 2–3 minutes per service. The `start_period: 180s` healthcheck grace reflects that. + +## Verify + +```bash +# Health +curl -s http://localhost:8001/health +curl -s http://localhost:8002/health + +# Embedding (OpenAI-compatible) +curl -s http://localhost:8001/v1/embeddings \ + -H "Content-Type: application/json" \ + -d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":["hello world"]}' | jq . + +# Reranker +curl -s http://localhost:8002/rerank \ + -H "Content-Type: application/json" \ + -d '{"model":"Qwen/Qwen3-Reranker-0.6B","query":"what is a cat","documents":["cats are mammals","dogs bark"]}' | jq . + +# Listed models +curl -s http://localhost:8001/v1/models | jq . +curl -s http://localhost:8002/v1/models | jq . +``` + +## Scaling knobs + +- **`EMBED_GPU_MEM_UTIL` / `RERANK_GPU_MEM_UTIL`** — fractions of **total** GPU VRAM each service reserves (not of free VRAM). Both services profile independently, so each slice must be large enough to fit that service's model + KV cache with no knowledge of the other. Setting them too low causes the second-to-start container to OOM on KV cache allocation with `Available KV cache memory: -X.XX GiB`. Default 0.40/0.40 (= 0.80 total) leaves ~20% GPU headroom and works cleanly for the 0.6B pair at 8k context; raise for 4B/8B variants or drop `max-model-len` if you need more room. +- **`EMBED_MAX_MODEL_LEN` / `RERANK_MAX_MODEL_LEN`** — lower to reduce KV-cache allocation if VRAM is tight. Qwen3 supports up to 32k natively. +- **Larger models** — Qwen3-Embedding/Reranker come in 0.6B / 4B / 8B. Swap `EMBED_MODEL` / `RERANK_MODEL` and bump the memory fractions accordingly. +- **Separate GPUs** — if contention hurts latency, split them: add a second `GPU_ID_RERANK` variable and point each service at its own device. (Requires a small compose edit; currently both share `${GPU_ID}`.) + +## Migrating off Infinity + +Once this stack is verified stable: + +1. Stop the infinity stack (`docker compose down` under `/opt/docker/compose/infinity/`). +2. Update consumers (AIPA agents, LibreChat RAG) to point at `:8001` for embeddings and `:8002` for rerank. +3. Delete `stacks/infinity/` from this workspace. diff --git a/stacks/vllm-qwen3/compose.yaml b/stacks/vllm-qwen3/compose.yaml new file mode 100644 index 0000000..4233333 --- /dev/null +++ b/stacks/vllm-qwen3/compose.yaml @@ -0,0 +1,133 @@ +# vLLM — Qwen3 Embedding + Reranker (one stack, two services). +# +# Replaces the unmaintained Infinity stack. vLLM runs one model per process, +# so this stack brings up two containers sharing a single GPU: +# +# vllm-embed — Qwen3-Embedding served as an OpenAI /v1/embeddings server +# vllm-rerank — Qwen3-Reranker served as a /rerank + /score server +# +# The reranker is a causal-LM checkpoint; --hf-overrides re-maps it to +# Qwen3ForSequenceClassification so vLLM's reranking endpoints work and the +# model only emits two class logits (no/yes) instead of the full 151k vocab. +# +# All tunables live in .env — edit that, not this file. +# +# Pre-download models to avoid first-run delay: +# HF_HOME=/tank/aimodels/huggingface hf download Qwen/Qwen3-Embedding-0.6B +# HF_HOME=/tank/aimodels/huggingface hf download Qwen/Qwen3-Reranker-0.6B + +services: + vllm-embed: + image: vllm/vllm-openai:${VLLM_VERSION} + container_name: vllm-embed + restart: unless-stopped + ipc: host + ports: + - "${EMBED_PORT}:8000" + volumes: + - /tank/aimodels/huggingface:/hfcache + environment: + - HF_HOME=/hfcache + - HF_HUB_CACHE=/hfcache/hub + - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-} + - VLLM_API_KEY=${API_KEY:-} + command: + - ${EMBED_MODEL} + - --served-model-name + - ${EMBED_MODEL} + - --runner + - pooling + - --host + - 0.0.0.0 + - --port + - "8000" + - --gpu-memory-utilization + - ${EMBED_GPU_MEM_UTIL} + - --max-model-len + - ${EMBED_MAX_MODEL_LEN} + - --dtype + - auto + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: + - "${GPU_ID}" + capabilities: + - gpu + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8000/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 180s + networks: + - tnet + labels: + - homepage.group=AI Systems + - homepage.name=vLLM Embed (Qwen3) + - homepage.icon=mdi-vector-arrange-below + - homepage.description=Qwen3 Embedding via vLLM (ana-ml2) + - homepage.href=http://10.250.50.54:${EMBED_PORT}/docs + + vllm-rerank: + image: vllm/vllm-openai:${VLLM_VERSION} + container_name: vllm-rerank + restart: unless-stopped + ipc: host + ports: + - "${RERANK_PORT}:8000" + volumes: + - /tank/aimodels/huggingface:/hfcache + environment: + - HF_HOME=/hfcache + - HF_HUB_CACHE=/hfcache/hub + - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-} + - VLLM_API_KEY=${API_KEY:-} + command: + - ${RERANK_MODEL} + - --served-model-name + - ${RERANK_MODEL} + - --runner + - pooling + - --hf-overrides + - '{"architectures":["Qwen3ForSequenceClassification"],"classifier_from_token":["no","yes"],"is_original_qwen3_reranker":true}' + - --host + - 0.0.0.0 + - --port + - "8000" + - --gpu-memory-utilization + - ${RERANK_GPU_MEM_UTIL} + - --max-model-len + - ${RERANK_MAX_MODEL_LEN} + - --dtype + - auto + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: + - "${GPU_ID}" + capabilities: + - gpu + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8000/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 180s + networks: + - tnet + labels: + - homepage.group=AI Systems + - homepage.name=vLLM Rerank (Qwen3) + - homepage.icon=mdi-sort-variant + - homepage.description=Qwen3 Reranker via vLLM (ana-ml2) + - homepage.href=http://10.250.50.54:${RERANK_PORT}/docs + +networks: + tnet: + name: traefik-net + external: true