Initial commit: PFI fleet inventory, stacks, tooling, and backup pipeline

Captures the full workspace state built up to this point: - CLAUDE.md + README.md describing conventions and the four-host fleet (ana-ml2, ana-docker, nh3-docker, esh-docker-vm). - Per-host notes under servers/<host>/ with ssh-target fallback files and latest system-details snapshots (two in-compose credential leaks scrubbed; the upstream compose files still need to move those to .env). - scripts/: server_inspect.sh (read-only remote diagnostic), refresh-server-info.sh (dir-driven discovery + snapshot capture with validation warnings), add-host.sh, sync-stacks.sh (pull compose/conf trees), deploy-stack.sh (push with per-file diff + prompt). - stacks/: canonical compose for backrest, beszel, dozzle, llama-swap, rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired infinity reference. All use the .env-driven + traefik-net + homepage label pattern. - configs/restic/ana-docker/: first resticprofile config + pre-backup hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates for the other three hosts to come. - docs/pfi/: general infrastructure reference carried over. - .gitignore excludes .env, stacks-mirror/, and assorted secret/state filenames to prevent re-leaks on later commits.
2026-04-20 14:29:48 -07:00
commit e376d0aec9
55 changed files with 9101 additions and 0 deletions
@@ -0,0 +1,117 @@
+# eshpfi-management
+
+Infrastructure management workspace for the PFI fleet (plus the ESH home-lab host). Tracks server state, canonical Docker Compose stacks, per-host configs, and the tooling that moves them around.
+
+See **[CLAUDE.md](CLAUDE.md)** for the full set of conventions and the rules Claude Code sessions follow when working here.
+
+## The fleet
+
+| Host | IP | Site | Role |
+|---|---|---|---|
+| ana-ml2 | `10.250.50.54` | Anaheim (`10.250.0.0/16`) | GPU / AI inference |
+| ana-docker | `10.250.50.70` | Anaheim | General-purpose Docker + cross-site hubs |
+| nh3-docker | `10.100.50.40` | NH3 (`10.100.0.0/16`) | General-purpose Docker (NH site) |
+| esh-docker-vm | `10.0.50.45` | ESH home lab (`esteban.net`) | Home-lab Docker (non-PFI scope) |
+
+Per-host snapshots of the running system live under `servers/<host>/system-details.txt`, refreshed via `scripts/refresh-server-info.sh`.
+
+## Layout
+
+```
+.
+├── CLAUDE.md                     # conventions; loaded by Claude Code sessions
+├── README.md                     # this file
+├── scripts/                      # workstation tooling
+│   ├── server_inspect.sh         # read-only diagnostic, runs on remote via stdin
+│   ├── refresh-server-info.sh    # pull fresh system-details.txt for one/all hosts
+│   ├── add-host.sh               # register a new server (writes servers/<name>/ssh-target)
+│   ├── sync-stacks.sh            # pull /opt/docker/{compose,conf}/ → stacks-mirror/
+│   └── deploy-stack.sh           # push stacks-mirror/<host>/<stack>/ with diff + prompt
+├── servers/                      # per-host notes + latest snapshot + ssh-target fallback
+│   └── <host>/
+│       ├── README.md
+│       ├── system-details.txt    # regenerate on demand
+│       └── ssh-target            # <ip> or <user>@<ip>, used when DNS fails
+├── stacks/                       # canonical compose files (source of truth)
+│   └── <stack>/
+│       ├── compose.yaml
+│       ├── .env.example
+│       └── README.md
+├── stacks-mirror/                # gitignored — live mirror from sync-stacks.sh
+├── configs/                      # host-level config files that aren't docker-compose
+│   └── restic/<host>/            # resticprofile configs + pre-backup hooks
+└── docs/                         # general reference (network, models, proxmox, etc.)
+    └── pfi/
+```
+
+## Current stacks
+
+**GPU (ana-ml2):**
+- `llama-swap` — GGUF model swapper via llama.cpp (port 9292)
+- `vllm-qwen3` — embeddings (8001) + reranker (8002) via vLLM
+
+**Anaheim non-GPU (ana-docker):**
+- `traefik`, `crowdsec`, `gitea`, `vaultwarden`, `synapse`, `seafile`, `searxng`, `openwebui`, `sillytavern`, `mailrise`, `rustdesk`, `dockge`, `it-tools`
+- Fleet services: `beszel` (metrics hub, port 8090), `dozzle-hub` (log viewer, 8088), `backrest` (restic UI, 9898)
+- Backup target: `rest-server-ana` on port 8000
+
+**NH3 (nh3-docker):**
+- `adguard`, `dockge`, plus Beszel/Dozzle agents
+
+**NH3 (Synology `10.100.50.50`):**
+- `rest-server-nh3` — restic backup target (port 8000)
+
+**ESH home lab (esh-docker-vm):**
+- `adguard`, `homeassistant` (macvlan), `esphome`, `mosquitto`, `paperless-ngx`, `pgadmin`, `calibre`, `calibre-web`, `drawio`, `traefik`, `homepage`, `uptime-kuma`, plus Beszel/Dozzle agents
+
+## Common tasks
+
+**Refresh one host's snapshot:**
+```bash
+scripts/refresh-server-info.sh ana-docker
+```
+
+**Refresh all hosts:**
+```bash
+scripts/refresh-server-info.sh all
+```
+
+**Add a new host:**
+```bash
+scripts/add-host.sh <name> <ip-or-user@ip>
+scripts/refresh-server-info.sh <name>
+```
+
+**Validate discovery (without hitting the network):**
+```bash
+scripts/refresh-server-info.sh --validate-only all
+```
+
+**Push a stack to a host (with diff + confirm):**
+```bash
+scripts/deploy-stack.sh <host> <stack>
+```
+
+**Pull every server's compose/conf trees into stacks-mirror/ (not committed — see `.gitignore`):**
+```bash
+scripts/sync-stacks.sh all
+```
+
+## Backup pipeline
+
+Backups are driven by per-host `resticprofile` configs under `configs/restic/<host>/`, scheduled via systemd timers on each host:
+
+- **Writes**: each host backs up to its site-local rest-server (`rest-server-ana` or `rest-server-nh3`), over HTTP basic-auth.
+- **Authentication**: shared `.htpasswd` file on both rest-servers, one entry per host; credentials stored in `/etc/restic/restic.env` on each client host.
+- **Encryption**: per-host client-side passphrase in `/etc/restic/password` (unique per repo; losing it = losing that host's backups).
+- **Visibility**: Backrest (`http://10.250.50.70:9898`) shows every repo for browsing/restore.
+- **Schedule**: backup at 01:00 daily, `forget` at 03:00 daily, weekly `check --read-data-subset 10%` on Sundays.
+- **Prune**: manual ceremony (rest-server runs with `--append-only`, which blocks destructive prune ops).
+- **Off-site**: cross-site rsync between the two rest-server data dirs is planned (not yet implemented).
+
+## Authoritative vs. mirror
+
+- **Authoritative:** files on each server under `/opt/docker/compose/<stack>/` and `/opt/docker/conf/<stack>/`.
+- **This workspace:** source-of-truth copies under `stacks/<name>/` (hand-curated), and a gitignored mirror under `stacks-mirror/` pulled by `sync-stacks.sh`.
+
+Edit in `stacks/`, push with `deploy-stack.sh`. Never commit `stacks-mirror/` — it can contain embedded plaintext secrets from upstream compose files that haven't been audited yet.