Initial commit: PFI fleet inventory, stacks, tooling, and backup pipeline

Captures the full workspace state built up to this point: - CLAUDE.md + README.md describing conventions and the four-host fleet (ana-ml2, ana-docker, nh3-docker, esh-docker-vm). - Per-host notes under servers/<host>/ with ssh-target fallback files and latest system-details snapshots (two in-compose credential leaks scrubbed; the upstream compose files still need to move those to .env). - scripts/: server_inspect.sh (read-only remote diagnostic), refresh-server-info.sh (dir-driven discovery + snapshot capture with validation warnings), add-host.sh, sync-stacks.sh (pull compose/conf trees), deploy-stack.sh (push with per-file diff + prompt). - stacks/: canonical compose for backrest, beszel, dozzle, llama-swap, rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired infinity reference. All use the .env-driven + traefik-net + homepage label pattern. - configs/restic/ana-docker/: first resticprofile config + pre-backup hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates for the other three hosts to come. - docs/pfi/: general infrastructure reference carried over. - .gitignore excludes .env, stacks-mirror/, and assorted secret/state filenames to prevent re-leaks on later commits.
2026-04-20 14:29:48 -07:00
commit e376d0aec9
55 changed files with 9101 additions and 0 deletions
@@ -0,0 +1,43 @@
+# beszel stack tunables. Copy to `.env` on each server before deploying.
+#
+#   cp .env.example .env
+#   # edit for this host
+#   docker compose up -d
+#
+# Same compose.yaml on both servers — COMPOSE_PROFILES picks the role.
+
+# Image version — pin for reproducibility (`latest` for edge)
+BESZEL_VERSION=latest
+
+# ------------------------------------------------------------------------
+# On ana-docker (hub + local agent):
+#   COMPOSE_PROFILES=hub,agent
+#   BESZEL_PORT=8090
+#   BESZEL_HUB_KEY=<copy from hub "Add system" dialog after first boot>
+#
+# On ana-ml2 (agent only):
+#   COMPOSE_PROFILES=agent
+#   BESZEL_HUB_KEY=<same key as above>
+# ------------------------------------------------------------------------
+
+COMPOSE_PROFILES=hub,agent
+
+# ---- Hub-only -----------------------------------------------------------
+
+# Host port for the web UI (container listens on 8090 internally).
+BESZEL_PORT=8090
+
+# ---- Agent-only ---------------------------------------------------------
+
+# Host port the agent listens on. The hub SSHes into agents over this port.
+BESZEL_AGENT_PORT=45876
+
+# Hub's SSH public key — paste from the hub UI on first run.
+# Grab it by clicking "Add System" → copy the key shown in the dialog.
+BESZEL_HUB_KEY=
+
+# Extra filesystems to track beyond the root mount, comma-separated.
+# Examples:
+#   on ana-ml2: /tank
+#   on ana-docker: /mnt/backup,/mnt/compose
+BESZEL_EXTRA_FS=
@@ -0,0 +1,88 @@
+# beszel
+
+Lightweight monitoring — CPU, memory, disk, network, and per-container stats for every Docker host, with alerts over email/webhook. Pairs with Dozzle (logs) on the same server.
+
+**Deploys to:**
+- **ana-docker** (hub + local agent) — UI at `http://10.250.50.70:8090`
+- **ana-ml2** (agent only) — listens on `10.250.50.54:45876`
+- **nh3-docker** (agent only, cross-site) — listens on `10.100.50.40:45876`
+
+Same compose.yaml on each host. Per-host `.env` sets `COMPOSE_PROFILES` to bring up the right combination. Each agent host is added individually in the hub UI.
+
+## How hub ↔ agent auth works
+
+Beszel uses SSH-key-based auth: the hub generates its own keypair on first boot, and each agent must be seeded with the hub's **public key** via the `KEY` env var. Agents listen on a port (default 45876); the hub pulls metrics by connecting to them with that key.
+
+Operator flow on first deploy:
+
+1. Bring up the **hub** on ana-docker with `BESZEL_HUB_KEY=` blank and the agent profile disabled.
+2. Open the UI, create the admin account, click **Add System** — Beszel shows the public key.
+3. Copy the key into `BESZEL_HUB_KEY` in the `.env` on both hosts.
+4. Re-deploy the hub with `COMPOSE_PROFILES=hub,agent` to add the local agent; deploy the agent on ana-ml2.
+5. Back in the UI, **Add System** with `host=127.0.0.1 port=45876` (local) and `host=10.250.50.54 port=45876` (ana-ml2).
+
+## Deploy — hub + local agent (ana-docker)
+
+```bash
+ssh ana-docker
+sudo mkdir -p /opt/docker/compose/beszel
+sudo chown $USER /opt/docker/compose/beszel
+cd /opt/docker/compose/beszel
+
+# scp compose.yaml + .env.example, then:
+cp .env.example .env
+# First pass — hub only, no key yet:
+#   COMPOSE_PROFILES=hub
+#   BESZEL_PORT=8090
+docker compose up -d
+
+# Open http://10.250.50.70:8090 → create admin → click "Add System" →
+# copy the displayed public key into BESZEL_HUB_KEY.
+
+# Second pass — add the local agent:
+#   COMPOSE_PROFILES=hub,agent
+#   BESZEL_EXTRA_FS=/mnt/backup,/mnt/compose
+docker compose up -d
+```
+
+## Deploy — agent (ana-ml2)
+
+```bash
+ssh ana-ml2
+sudo mkdir -p /opt/docker/compose/beszel
+sudo chown $USER /opt/docker/compose/beszel
+cd /opt/docker/compose/beszel
+
+# scp the same compose.yaml + .env.example, then:
+cp .env.example .env
+# Edit to:
+#   COMPOSE_PROFILES=agent
+#   BESZEL_HUB_KEY=<same key as the hub>
+#   BESZEL_EXTRA_FS=/tank
+
+docker compose up -d
+docker compose logs -f
+```
+
+Then in the hub UI, **Add System** with `host=10.250.50.54`, `port=45876`.
+
+## Verify
+
+```bash
+# Hub health
+curl -s http://10.250.50.70:8090/api/health
+
+# Agent reachable
+ssh ana-docker 'nc -zv 10.250.50.54 45876'
+
+# Local agent reachable from hub container
+docker exec beszel nc -zv host.docker.internal 45876
+```
+
+## Sizing / impact
+
+The agent is ~10 MB RAM and negligible CPU — runs fine alongside anything on ana-ml2 including GPU workloads. Host-mode networking means it has no port conflicts with other stacks as long as `BESZEL_AGENT_PORT` stays unique.
+
+## Alerts
+
+Configured inside the hub UI (Settings → Notifications). Supports email (SMTP), Gotify, ntfy, Discord, Slack, and generic webhooks. Alert rules attach to per-system or global thresholds (CPU, memory, disk, container down, etc.).
@@ -0,0 +1,63 @@
+# Beszel — lightweight server/container monitoring.
+#
+# Hub: single web UI with the SQLite store. Agents: per-host metric collectors
+# that the hub pulls from over SSH.
+#
+# Multi-host layout via compose profiles:
+#   COMPOSE_PROFILES=hub          → hub only        (ana-docker)
+#   COMPOSE_PROFILES=hub,agent    → hub + local agent on the same host
+#   COMPOSE_PROFILES=agent        → agent only      (ana-ml2)
+#
+# The agent uses network_mode: host so it sees real host CPU/mem/net/disk
+# counters rather than container-scoped ones — that's why it can't share
+# the tnet network with the hub.
+#
+# All tunables live in .env — edit that, not this file.
+
+services:
+  beszel:
+    image: henrygd/beszel:${BESZEL_VERSION}
+    container_name: beszel
+    profiles: [hub]
+    restart: unless-stopped
+    ports:
+      - "${BESZEL_PORT}:8090"
+    volumes:
+      - beszel_data:/beszel_data
+    healthcheck:
+      test: ["CMD", "wget", "-qO-", "http://localhost:8090/api/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 15s
+    networks:
+      - tnet
+    labels:
+      - homepage.group=PFI-ANA
+      - homepage.name=Beszel
+      - homepage.icon=mdi-chart-line
+      - homepage.description=Server + container monitoring
+      - homepage.href=http://10.250.50.70:${BESZEL_PORT}
+
+  beszel-agent:
+    image: henrygd/beszel-agent:${BESZEL_VERSION}
+    container_name: beszel-agent
+    profiles: [agent]
+    restart: unless-stopped
+    network_mode: host
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - beszel_agent_data:/var/lib/beszel-agent
+    environment:
+      - PORT=${BESZEL_AGENT_PORT:-45876}
+      - KEY=${BESZEL_HUB_KEY}
+      - EXTRA_FILESYSTEMS=${BESZEL_EXTRA_FS:-}
+
+volumes:
+  beszel_data:
+  beszel_agent_data:
+
+networks:
+  tnet:
+    name: traefik-net
+    external: true