Initial commit: PFI fleet inventory, stacks, tooling, and backup pipeline

Captures the full workspace state built up to this point: - CLAUDE.md + README.md describing conventions and the four-host fleet (ana-ml2, ana-docker, nh3-docker, esh-docker-vm). - Per-host notes under servers/<host>/ with ssh-target fallback files and latest system-details snapshots (two in-compose credential leaks scrubbed; the upstream compose files still need to move those to .env). - scripts/: server_inspect.sh (read-only remote diagnostic), refresh-server-info.sh (dir-driven discovery + snapshot capture with validation warnings), add-host.sh, sync-stacks.sh (pull compose/conf trees), deploy-stack.sh (push with per-file diff + prompt). - stacks/: canonical compose for backrest, beszel, dozzle, llama-swap, rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired infinity reference. All use the .env-driven + traefik-net + homepage label pattern. - configs/restic/ana-docker/: first resticprofile config + pre-backup hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates for the other three hosts to come. - docs/pfi/: general infrastructure reference carried over. - .gitignore excludes .env, stacks-mirror/, and assorted secret/state filenames to prevent re-leaks on later commits.
2026-04-20 14:29:48 -07:00
commit e376d0aec9
55 changed files with 9101 additions and 0 deletions
@@ -0,0 +1,88 @@
+# beszel
+
+Lightweight monitoring — CPU, memory, disk, network, and per-container stats for every Docker host, with alerts over email/webhook. Pairs with Dozzle (logs) on the same server.
+
+**Deploys to:**
+- **ana-docker** (hub + local agent) — UI at `http://10.250.50.70:8090`
+- **ana-ml2** (agent only) — listens on `10.250.50.54:45876`
+- **nh3-docker** (agent only, cross-site) — listens on `10.100.50.40:45876`
+
+Same compose.yaml on each host. Per-host `.env` sets `COMPOSE_PROFILES` to bring up the right combination. Each agent host is added individually in the hub UI.
+
+## How hub ↔ agent auth works
+
+Beszel uses SSH-key-based auth: the hub generates its own keypair on first boot, and each agent must be seeded with the hub's **public key** via the `KEY` env var. Agents listen on a port (default 45876); the hub pulls metrics by connecting to them with that key.
+
+Operator flow on first deploy:
+
+1. Bring up the **hub** on ana-docker with `BESZEL_HUB_KEY=` blank and the agent profile disabled.
+2. Open the UI, create the admin account, click **Add System** — Beszel shows the public key.
+3. Copy the key into `BESZEL_HUB_KEY` in the `.env` on both hosts.
+4. Re-deploy the hub with `COMPOSE_PROFILES=hub,agent` to add the local agent; deploy the agent on ana-ml2.
+5. Back in the UI, **Add System** with `host=127.0.0.1 port=45876` (local) and `host=10.250.50.54 port=45876` (ana-ml2).
+
+## Deploy — hub + local agent (ana-docker)
+
+```bash
+ssh ana-docker
+sudo mkdir -p /opt/docker/compose/beszel
+sudo chown $USER /opt/docker/compose/beszel
+cd /opt/docker/compose/beszel
+
+# scp compose.yaml + .env.example, then:
+cp .env.example .env
+# First pass — hub only, no key yet:
+#   COMPOSE_PROFILES=hub
+#   BESZEL_PORT=8090
+docker compose up -d
+
+# Open http://10.250.50.70:8090 → create admin → click "Add System" →
+# copy the displayed public key into BESZEL_HUB_KEY.
+
+# Second pass — add the local agent:
+#   COMPOSE_PROFILES=hub,agent
+#   BESZEL_EXTRA_FS=/mnt/backup,/mnt/compose
+docker compose up -d
+```
+
+## Deploy — agent (ana-ml2)
+
+```bash
+ssh ana-ml2
+sudo mkdir -p /opt/docker/compose/beszel
+sudo chown $USER /opt/docker/compose/beszel
+cd /opt/docker/compose/beszel
+
+# scp the same compose.yaml + .env.example, then:
+cp .env.example .env
+# Edit to:
+#   COMPOSE_PROFILES=agent
+#   BESZEL_HUB_KEY=<same key as the hub>
+#   BESZEL_EXTRA_FS=/tank
+
+docker compose up -d
+docker compose logs -f
+```
+
+Then in the hub UI, **Add System** with `host=10.250.50.54`, `port=45876`.
+
+## Verify
+
+```bash
+# Hub health
+curl -s http://10.250.50.70:8090/api/health
+
+# Agent reachable
+ssh ana-docker 'nc -zv 10.250.50.54 45876'
+
+# Local agent reachable from hub container
+docker exec beszel nc -zv host.docker.internal 45876
+```
+
+## Sizing / impact
+
+The agent is ~10 MB RAM and negligible CPU — runs fine alongside anything on ana-ml2 including GPU workloads. Host-mode networking means it has no port conflicts with other stacks as long as `BESZEL_AGENT_PORT` stays unique.
+
+## Alerts
+
+Configured inside the hub UI (Settings → Notifications). Supports email (SMTP), Gotify, ntfy, Discord, Slack, and generic webhooks. Alert rules attach to per-system or global thresholds (CPU, memory, disk, container down, etc.).