eshpfi-management

Infrastructure management workspace for the PFI fleet (plus the ESH home-lab host). Tracks server state, canonical Docker Compose stacks, per-host configs, and the tooling that moves them around.

See CLAUDE.md for the full set of conventions and the rules Claude Code sessions follow when working here.

The fleet

Docker hosts:

Host	IP	Site	Role
ana-ml2	10.250.50.54	Anaheim (10.250.0.0/16)	GPU / AI inference (bare metal)
ana-docker	10.250.50.70	Anaheim	General-purpose Docker + cross-site hubs (VM on pfi-pve)
nh3-docker	10.100.50.40	NH3 (10.100.0.0/16)	General-purpose Docker (VM on nh3-pve)
esh-docker-vm	10.0.50.45	ESH home lab (esteban.net)	Home-lab Docker (VM on esh-pve, non-PFI scope)

Proxmox hypervisors (tracked for inventory; not Docker targets):

Host	IP	Site	Role
pfi-pve	10.250.250.31	Anaheim	Proxmox VE (188 GB / Xeon Silver 4310)
nh3-pve	10.100.250.60	NH3	Proxmox VE (62 GB / i9-13900H)
esh-pve	10.0.250.35	ESH home lab	Proxmox VE (62 GB / i9-13900H)
esh-pve-nas	10.0.50.55	ESH home lab	Proxmox VE, storage-dedicated (125 GB / Xeon W-1250)

Per-host snapshots of the running system live under servers/<host>/system-details.txt, refreshed via scripts/refresh-server-info.sh.

Layout

.
├── CLAUDE.md                     # conventions; loaded by Claude Code sessions
├── README.md                     # this file
├── scripts/                      # workstation tooling
│   ├── server_inspect.sh         # read-only diagnostic, runs on remote via stdin
│   ├── refresh-server-info.sh    # pull fresh system-details.txt for one/all hosts
│   ├── add-host.sh               # register a new server (writes servers/<name>/ssh-target)
│   ├── sync-stacks.sh            # pull /opt/docker/{compose,conf}/ → stacks-mirror/
│   └── deploy-stack.sh           # push stacks-mirror/<host>/<stack>/ with diff + prompt
├── servers/                      # per-host notes + latest snapshot + ssh-target fallback
│   └── <host>/
│       ├── README.md
│       ├── system-details.txt    # regenerate on demand
│       └── ssh-target            # <ip> or <user>@<ip>, used when DNS fails
├── stacks/                       # canonical compose files (source of truth)
│   └── <stack>/
│       ├── compose.yaml
│       ├── .env.example
│       └── README.md
├── stacks-mirror/                # gitignored — live mirror from sync-stacks.sh
├── configs/                      # host-level config files that aren't docker-compose
│   └── restic/<host>/            # resticprofile configs + pre-backup hooks
└── docs/                         # general reference (network, models, proxmox, etc.)
    └── pfi/

Current stacks

GPU (ana-ml2):

• llama-swap — GGUF model swapper via llama.cpp (port 9292)
• vllm-qwen3 — embeddings (8001) + reranker (8002) via vLLM

Anaheim non-GPU (ana-docker):

• traefik, crowdsec, gitea, vaultwarden, synapse, seafile, searxng, openwebui, sillytavern, mailrise, rustdesk, dockge, it-tools
• Fleet services: beszel (metrics hub, port 8090), dozzle-hub (log viewer, 8088), backrest (restic UI, 9898)
• Backup target: rest-server-ana on port 8000

NH3 (nh3-docker):

• adguard, dockge, plus Beszel/Dozzle agents

NH3 (Synology 10.100.50.50):

• rest-server-nh3 — restic backup target (port 8000)

ESH home lab (esh-docker-vm):

• adguard, homeassistant (macvlan), esphome, mosquitto, paperless-ngx, pgadmin, calibre, calibre-web, drawio, traefik, homepage, uptime-kuma, plus Beszel/Dozzle agents

Common tasks

Refresh one host's snapshot:

scripts/refresh-server-info.sh ana-docker

Refresh all hosts:

scripts/refresh-server-info.sh all

Add a new host:

scripts/add-host.sh <name> <ip-or-user@ip>
scripts/refresh-server-info.sh <name>

Validate discovery (without hitting the network):

scripts/refresh-server-info.sh --validate-only all

Push a stack to a host (with diff + confirm):

scripts/deploy-stack.sh <host> <stack>

Pull every server's compose/conf trees into stacks-mirror/ (not committed — see .gitignore):

scripts/sync-stacks.sh all

Backup pipeline

Backups are driven by per-host resticprofile configs under configs/restic/<host>/, scheduled via systemd timers on each host:

• Writes: each host backs up to its site-local rest-server (rest-server-ana or rest-server-nh3), over HTTP basic-auth.
• Authentication: shared .htpasswd file on both rest-servers, one entry per host; credentials stored in /etc/restic/restic.env on each client host.
• Encryption: per-host client-side passphrase in /etc/restic/password (unique per repo; losing it = losing that host's backups).
• Visibility: Backrest (http://10.250.50.70:9898) shows every repo for browsing/restore.
• Schedule: backup at 01:00 daily, forget at 03:00 daily, weekly check --read-data-subset 10% on Sundays.
• Prune: manual ceremony (rest-server runs with --append-only, which blocks destructive prune ops).
• Off-site: cross-site rsync between the two rest-server data dirs is planned (not yet implemented).

Authoritative vs. mirror

• Authoritative: files on each server under /opt/docker/compose/<stack>/ and /opt/docker/conf/<stack>/.
• This workspace: source-of-truth copies under stacks/<name>/ (hand-curated), and a gitignored mirror under stacks-mirror/ pulled by sync-stacks.sh.

Edit in stacks/, push with deploy-stack.sh. Never commit stacks-mirror/ — it can contain embedded plaintext secrets from upstream compose files that haven't been audited yet.