vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b842212b0637b4b11e02eccb0064e4836f6efdd0
esh-pfi-infrastructure/README.md
T
vh 921891b27a scripts: network-discovery tooling for FortiGate + UniFi 
Three scripts that surface hosts on the fleet's networks that aren't
already tracked under servers/*/. Goal: spot servers that need management
coverage (inventory, backup, monitoring) without wandering the LAN by
hand.

  discover-fortigate.sh  SSH to a FortiGate admin, run
                         `execute dhcp lease-list all`, emit TSV
                         (IP, MAC, hostname, vdom, source).

                         SSH was picked over the REST API for now
                         because it needs no API-token plumbing. The
                         parser is defensive about FortiOS output
                         format drift (multiple VDOM sections,
                         optional hostname). API variant can replace
                         it when the extra robustness is worth the
                         token setup.

  discover-unifi.sh      Cookie-auth REST call against a UniFi
                         Controller. Tries /api/auth/login (UniFi OS
                         / UDM / Cloud Key Gen2+) first; falls back
                         to legacy /api/login for self-hosted
                         controllers. Output is the same TSV shape
                         as the FortiGate script so the two mix.

                         Needs UNIFI_USER / UNIFI_PASS env and jq.

  discover-gaps.sh       Consumes one or more TSVs from the sources
                         above. Builds the set of managed IPs from
                         servers/*/ssh-target (plus a grep of README
                         files for documented IPs) and prints any
                         discovered IPs not in that set.

                         Exit code is 1 if gaps found — suitable for
                         cron alerting.

Common pipeline:
  scripts/discover-fortigate.sh ana-fw.phasefinal.com > /tmp/ana.tsv
  scripts/discover-fortigate.sh nh3-gw.phasefinal.com > /tmp/nh3.tsv
  UNIFI_USER=admin UNIFI_PASS=… scripts/discover-unifi.sh esh-uc.esteban.net > /tmp/esh.tsv
  scripts/discover-gaps.sh /tmp/ana.tsv /tmp/nh3.tsv /tmp/esh.tsv

First-time use probably needs:
  - SSH access configured to each FortiGate (admin login, key preferred)
  - UniFi user with read access (the built-in API read-only role works)
  - `jq` installed on the runner (for UniFi script)
2026-04-21 11:51:00 -07:00

166 lines
8.4 KiB
Markdown
Raw Blame History

# eshpfi-management
Infrastructure management workspace for the PFI fleet (plus the ESH home-lab host). Tracks server state, canonical Docker Compose stacks, per-host configs, and the tooling that moves them around.
See **[CLAUDE.md](CLAUDE.md)** for the full set of conventions and the rules Claude Code sessions follow when working here.
## The fleet
**Docker hosts:**
| Host | IP | Site | Role |
|---|---|---|---|
| ana-ml2 | `10.250.50.54` | Anaheim (`10.250.0.0/16`) | GPU / AI inference (bare metal) |
| ana-docker | `10.250.50.70` | Anaheim | General-purpose Docker + cross-site hubs (VM on pfi-pve) |
| nh3-docker | `10.100.50.40` | NH3 (`10.100.0.0/16`) | General-purpose Docker (VM on nh3-pve) |
| esh-docker-vm | `10.0.50.45` | ESH home lab (`esteban.net`) | Home-lab Docker (VM on esh-pve, non-PFI scope) |
| vm-esh-nas | `10.0.50.154` | ESH home lab | NAS-adjacent Docker, NFS-mounted shares (VM on esh-pve-nas, non-PFI scope) |
**Proxmox hypervisors** (tracked for inventory; not Docker targets):
| Host | IP | Site | Role |
|---|---|---|---|
| pfi-pve | `10.250.250.31` | Anaheim | Proxmox VE (188 GB / Xeon Silver 4310) |
| nh3-pve | `10.100.250.60` | NH3 | Proxmox VE (62 GB / i9-13900H) |
| esh-pve | `10.0.250.35` | ESH home lab | Proxmox VE (62 GB / i9-13900H) |
| esh-pve-nas | `10.0.50.55` | ESH home lab | Proxmox VE, storage-dedicated (125 GB / Xeon W-1250) |
Per-host snapshots of the running system live under `servers/<host>/system-details.txt`, refreshed via `scripts/refresh-server-info.sh`.
## Layout
```
.
├── CLAUDE.md # conventions; loaded by Claude Code sessions
├── README.md # this file
├── scripts/ # workstation tooling
│ ├── server_inspect.sh # read-only diagnostic, runs on remote via stdin
│ ├── proxmox_inspect.sh # Proxmox-aware probe (VMs, LXCs, storage, backup coverage)
│ ├── refresh-server-info.sh # pull fresh system-details.txt for one/all hosts
│ ├── refresh-proxmox-info.sh # pull fresh proxmox-details.txt for one/all PVE nodes
│ ├── add-host.sh # register a new server (writes servers/<name>/ssh-target)
│ ├── sync-stacks.sh # pull /opt/docker/{compose,conf}/ → stacks-mirror/
│ ├── deploy-stack.sh # push stacks-mirror/<host>/<stack>/ with diff + prompt
│ ├── discover-fortigate.sh # DHCP lease list from a FortiGate via SSH
│ ├── discover-unifi.sh # client list from a UniFi Controller via REST
│ └── discover-gaps.sh # find IPs in discovery TSVs not tracked in servers/
├── servers/ # per-host notes + latest snapshot + ssh-target fallback
│ └── <host>/
│ ├── README.md
│ ├── system-details.txt # regenerate on demand
│ └── ssh-target # <ip> or <user>@<ip>, used when DNS fails
├── stacks/ # canonical compose files (source of truth)
│ └── <stack>/
│ ├── compose.yaml
│ ├── .env.example
│ └── README.md
├── stacks-mirror/ # gitignored — live mirror from sync-stacks.sh
├── configs/ # host-level config files that aren't docker-compose
│ ├── homepage/ # canonical config for the fleet dashboard (on esh-docker-vm)
│ └── restic/<host>/ # resticprofile configs + pre-backup hooks
└── docs/ # general reference (network, models, proxmox, etc.)
└── pfi/
```
## Current stacks
**GPU (ana-ml2):**
- `llama-swap` — GGUF model swapper via llama.cpp (port 9292)
- `vllm-qwen3` — embeddings (8001) + reranker (8002) via vLLM
**Anaheim non-GPU (ana-docker):**
- `traefik`, `crowdsec`, `gitea`, `vaultwarden`, `synapse`, `seafile`, `searxng`, `openwebui`, `sillytavern`, `mailrise`, `rustdesk`, `dockge`, `it-tools`
- (`mattermost` retired 2026-04-21 — compose dir may still linger, containers gone)
- Fleet services: `beszel` (metrics hub, port 8090), `dozzle-hub` (log viewer, 8088), `backrest` (restic UI, 9898)
- Backup target: `rest-server-ana` on port 8000
**GPU ana-ml2 (non-canonical for now):**
- `comfyui`, `kokoro`, `parakeet`, `vibevoice` alongside the canonical `llama-swap` + `vllm-qwen3`
**NH3 (nh3-docker):**
- `adguard`, `dockge`, plus Beszel/Dozzle agents
**NH3 (Synology `10.100.50.50`):**
- `rest-server-nh3` — restic backup target (port 8000)
**ESH home lab (esh-docker-vm):**
- `adguard`, `homeassistant` (macvlan), `esphome`, `mosquitto`, `paperless-ngx`, `pgadmin`, `calibre-web-automated`, `drawio`, `traefik`, `homepage`, `uptime-kuma`, plus Beszel/Dozzle agents
**ESH home lab (vm-esh-nas):**
- `filezilla` (web UI on port 5800), `dockge`, plus Beszel/Dozzle agents. Mounts `/mnt/{share,music,books,media}` from the Debian NAS at 10.0.50.50.
## Common tasks
**Refresh one host's snapshot:**
```bash
scripts/refresh-server-info.sh ana-docker
```
**Refresh all hosts:**
```bash
scripts/refresh-server-info.sh all
```
**Refresh all Proxmox nodes (separate flow — captures VM/LXC/backup-coverage):**
```bash
scripts/refresh-proxmox-info.sh all
```
**Add a new host:**
```bash
scripts/add-host.sh <name> <ip-or-user@ip>
scripts/refresh-server-info.sh <name>
```
**Validate discovery (without hitting the network):**
```bash
scripts/refresh-server-info.sh --validate-only all
```
**Push a stack to a host (with diff + confirm):**
```bash
scripts/deploy-stack.sh <host> <stack>
```
**Pull every server's compose/conf trees into stacks-mirror/ (not committed — see `.gitignore`):**
```bash
scripts/sync-stacks.sh all
```
## Backup pipeline
Backups are driven by per-host `resticprofile` configs under `configs/restic/<host>/`, scheduled via systemd timers on each host:
- **Writes**: each host backs up to its site-local rest-server (`rest-server-ana` or `rest-server-nh3`), over HTTP basic-auth.
- **Authentication**: shared `.htpasswd` file on both rest-servers, one entry per host; credentials stored in `/etc/restic/restic.env` on each client host.
- **Encryption**: per-host client-side passphrase in `/etc/restic/password` (unique per repo; losing it = losing that host's backups).
- **Visibility**: Backrest (`http://10.250.50.70:9898`) shows every repo for browsing/restore.
- **Schedule**: backup at 01:00 daily, `forget` at 03:00 daily, weekly `check --read-data-subset 10%` on Sundays.
- **Prune**: manual ceremony (rest-server runs with `--append-only`, which blocks destructive prune ops).
- **Off-site**: cross-site rsync between the two rest-server data dirs is planned (not yet implemented).
### Coverage status (as of 2026-04-20)
Goal: **every Docker host + configs + every database** covered, not just VM images.
| Layer | State |
|---|---|
| VM-level (Proxmox vzdump) | ✅ All running guests covered across pfi-pve / nh3-pve / esh-pve-nas; esh-pve has VMID 108 uncovered |
| ana-docker restic (host files + DBs) | ✅ `configs/restic/ana-docker/` with pre-backup hooks for synapse / seafile / vaultwarden |
| ana-ml2 restic | ❌ **Bare metal — no vzdump**, no restic yet. Highest-priority gap |
| nh3-docker restic | ✅ Light — no DB hooks needed |
| esh-docker-vm restic | ✅ With DB hooks for paperless-postgres (external), home-assistant + pgadmin + uptime-kuma (host-side sqlite3), calibre-web-automated (in-container sqlite3) |
| vm-esh-nas restic | ✅ Light — NFS mounts explicitly excluded |
| nh3-dev (workstation) restic | ✅ `/home/lkraven` + `/etc` with language-toolchain and build-output excludes |
| DB dumps: mattermost, openwebui, gitea, beszel-hub | ❌ Pre-backup hooks not written yet |
| Cross-site redundancy | ❌ rsync between rest-server-ana ↔ rest-server-nh3 planned |
| Prune ceremony | ❌ `scripts/restic-prune.sh` planned (rest-server `--append-only` blocks direct prune) |
| offen/docker-volume-backup sidecars on esh-docker-vm | ❌ Remove once restic proves itself (~1 week of clean runs) |
## Authoritative vs. mirror
- **Authoritative:** files on each server under `/opt/docker/compose/<stack>/` and `/opt/docker/conf/<stack>/`.
- **This workspace:** source-of-truth copies under `stacks/<name>/` (hand-curated), and a gitignored mirror under `stacks-mirror/` pulled by `sync-stacks.sh`.
Edit in `stacks/`, push with `deploy-stack.sh`. Never commit `stacks-mirror/` — it can contain embedded plaintext secrets from upstream compose files that haven't been audited yet.
Reference in New Issue View Git Blame Copy Permalink