End-to-end runbook for standing up Proxmox Backup Server across the
fleet. Path A architecture: single primary at ANA, one-way sync to NH3
for disaster recovery. All 5 hypervisors (pfi-pve, nh3-pve, esh-pve,
esh-pve-nas, sfsrv-ana) migrate from local-dump vzdump to PBS-ANA.
Key decisions captured in the runbook:
- PBS in a Debian VM (not LXC) for clean capability model.
- PBS-ANA on pfi-pve, datastore via NFS from 10.250.50.50 —
separates backup data from hypervisor boot disk.
- PBS-NH3 on nh3-pve with local storage (independent failure
domain from ANA).
- Dedicated fleet-vzdump API token; read-only sync token for
PBS-NH3's pull job.
- sfsrv-ana specifically goes from zero backup coverage to full
vzdump coverage in Phase 3.
9 phases, each self-contained with a done-state and rollback
posture. User can stop between phases without leaving the fleet in a
bad state.
STATUS.md: added item 6b tracking this deployment. Original item 6
(cross-site rsync) now scoped to restic-only since PBS handles the
VM-image cross-site redundancy directly.
Forget against an --append-only rest-server fails every night (delete
ops blocked). The resulting daily failure cluttered service status and
logs without ever actually retiring old snapshots. Schedule is now
removed from the forget block in all six profiles; the keep-daily /
keep-weekly / keep-monthly / keep-yearly policy remains so manual
invocations (during prune ceremonies, when --append-only is
temporarily off) honor the intended retention.
Files:
configs/restic/ana-docker/profiles.yaml
configs/restic/ana-ml2/profiles.yaml
configs/restic/nh3-docker/profiles.yaml
configs/restic/esh-docker-vm/profiles.yaml
configs/restic/vm-esh-nas/profiles.yaml
configs/restic/nh3-dev/profiles.yaml
Each file has an inline comment marking why the schedule was dropped
so a future reader doesn't re-add it thinking it was an oversight.
STATUS.md: removed the "install Backrest nightly-restart timer" line
item. User confirmed the UI timeout hits even at startup, so periodic
restart wouldn't actually help. Root cause remains deferred.
BatchMode=yes (which the scripts set) implies strict host key checking
and refuses to prompt — so first-time SSH to a host that isn't in
known_hosts fails with "Host key verification failed". Every new host
we register needs a manual `ssh <host>` round-trip first to store the
key before the refresh scripts can reach it.
accept-new fixes that: unknown hosts are auto-accepted into
known_hosts on first connect; subsequent key CHANGES still fail loudly
(as they should — that'd be a MITM signal).
Matches the pattern already used by deploy-stack.sh.
Affects only refresh-server-info.sh and refresh-proxmox-info.sh;
deploy-stack.sh + sync-stacks.sh use their own targets that may or may
not want the same treatment (leaving alone for now).
Initial framing was wrong. PFI runs these under a managed-hosting
agreement: SSH, OS ops, backups are all PFI's responsibility. Hardware
and data belong to the client.
Changes:
- ssh-target files added for sfsrv-ana (root@10.250.250.115 — same
pattern as other PVE nodes) and sf-ana-container
(lkraven@10.250.150.100 guess, adjust if different user).
- sf-r630 still lacks an ssh-target — the OS-side LAN IP isn't in
FortiGate DHCP (static config somewhere). Will fill in once
identified; README flags that gap.
- READMEs rewritten: dropped "tenant-scoped" / "not SSH-managed"
language, added "client context" section that explains the
managed-hosting relationship. Backup coverage now listed as
planned rather than blocked on tenant coordination.
- CLAUDE.md fleet table: SF rows re-labeled "SureFire client
(PFI-managed)". Placement-rules section updated to note that
SF hosts are first-class PFI-ops targets, just client-owned.
- Memory (project_surefire_tenant.md) rewritten to reflect
managed-services reality + hosts-file entries needed for name
resolution since these aren't in PFI DNS.
Snapshot of what's in place and what's outstanding as of end of
2026-04-20/21 session. Grouped by urgency (red/orange/yellow/green/blue)
so a glance tells you what's next regardless of who's picking it up.
Notable open items:
- Backrest esh-docker-vm URI mismatch (still pointed at NH3 Synology
instead of rest-server-ana)
- ssh-target verification on the 9 newly-added host entries
- Forget schedules need patching (fail nightly against --append-only)
- ~6 secrets captured in this session's transcripts need rotation
- SureFire tenant backup plan pending decision
Lists session milestones (homepage reorg, 6/6 restic coverage, discovery
scripts, CWA migration, 9 host registrations, etc.) and memory
pointers so future sessions have context without re-reading the full
chat log.
Six PFI VMs/LXCs previously known only via proxmox_inspect.sh —
covered by vzdump but not in servers/, so operational context
(roles, backup posture, ssh target) was missing:
pfi-ana-webhost (VMID 110) — web workload
ana-filebot (LXC 112) — file-task automation
pfi-pteradactyl (VMID 107) — Pterodactyl game panel
pfi-tacticalrmm (VMID 111) — TacticalRMM remote-management
pfi-postgres (VMID 105) — shared Postgres (vaultwarden/gitea/
paperless backends)
ana-wg (LXC 113) — WireGuard VPN gateway
Plus three SureFire tenant hosts at the Anaheim colo:
sfsrv-ana — tenant Proxmox hypervisor (10.250.250.115:8006)
sf-ana-container — container workload on that Proxmox
sf-r630 — physical R630 (iDRAC 10.250.250.110 for PFI-side
hardware mgmt; OS is tenant-scoped)
Each server dir has README + ssh-target where applicable. SureFire
entries explicitly document tenancy scope: PFI provides hosting,
SureFire owns the OS; management actions need tenant coordination.
SureFire hosts have no ssh-target by default.
Homepage Infra - ANA gains two new cards:
- SFsrv-ANA (https://10.250.250.115:8006, si-proxmox icon)
- SF-R630-iDRAC (https://10.250.250.110, si-dell icon)
PFI-ANA-ML2 BMC gained an href since it has a usable web UI.
CLAUDE.md fleet table extended with all 9 new rows. Placement-rules
section notes the SureFire tenant boundary.
Memory: new project_surefire_tenant.md so future sessions know sf-*
hosts are tenant-scoped by default.
First real run surfaced 31 gap rows, ~20 of which were noise. These
changes reduce the output to actionable signal.
1. discover-unifi: /ea/devices now filters out
- IPs outside the fleet LAN range (UDM's WAN IP appearing as a
"device", ISP uplink records with public IPs)
- UDM self-records (isConsole=true, or IP matches wans[].ipv4)
- UCI records (UniFi Cable Internet = ISP modem tracking)
LAN filter regex defaults to ^10\. (matches 10.0.0.0/8); override
via UNIFI_LAN_FILTER env var if you run other private ranges.
2. discover-gaps: new --ignore-unifi flag drops rows where the final
SOURCE column starts with "unifi:". Useful for "show me servery
things to manage, not the fleet's network hardware."
3. discover-gaps: known-IP set now pulls IPs from
servers/*/proxmox-details.txt AND servers/*/system-details.txt in
addition to README.md and ssh-target. Consequence: VMs tracked by
proxmox_inspect.sh are automatically counted as known without
needing a separate servers/<vmname>/ dir. Also strips meaningless
addresses (127.*, 0.0.0.0, 169.254.*) so they can't false-positive
a "known" match.
4. MAC normalization: both discover-fortigate and discover-unifi now
emit xx:xx:xx:xx:xx:xx lowercase. Previously FortiGate used colon
format, UniFi used no-separator uppercase — same MAC looked
different per source. Fortigate does tolower() in awk; UniFi uses
a shared jq `norm_mac` function.
Raw dumps of /ea/hosts and /ea/devices surfaced the actual JSON:
- /ea/hosts: LAN IP isn't at top-level ipAddress (that's WAN public);
it's buried in reportedState.ipAddrs[] mixed with WAN + link-local.
Have to pick the first RFC1918 entry that ISN'T also a WAN interface
IP (reportedState.wans[].ipv4). Name/mac/model all live under
reportedState.{hostname,mac,hardware.shortname}.
- /ea/devices: outer records are per-host wrappers; real AP/switch
records are in the nested `devices` array with top-level `ip`, `mac`,
`name`, `model` fields. Previous parser was reading the wrapper and
getting all `-`.
Reorder all TSV outputs so IP is column 1 — makes discover-gaps.sh
work uniformly against both FortiGate and UniFi sources. Sites TSV
dropped its IP slot since sites have no meaningful IP (metadata only).
Verified against the real payloads the user captured: ESH-UDMPM now
surfaces as 10.0.0.1 (LAN) instead of 192.168.200.111 (WAN2, RFC1918
but excluded via the wans cross-check). A sample device record
(E7-ESH-Media at 10.0.250.176) flattens correctly into a single TSV row.
Rewrite to use Ubiquiti's public cloud API at api.ui.com instead of
logging into individual controllers via session cookies. Benefits:
- One API key covers every UniFi OS device on the account (no
per-controller login logic, no cookie jar lifecycle).
- Read-only by design (auth keys are scoped).
- Works across sites transparently.
Three endpoints wired up: hosts (controllers / Cloud Keys), sites,
and devices (APs / switches). Each emits a distinct TSV shape so the
output can be concatenated and still parsed.
`all` mode runs all three and prints section markers on stderr so
the stdout stream stays clean TSV suitable for discover-gaps.sh.
Pagination handled via nextToken. Rate limit not enforced locally;
Ubiquiti documents generous defaults for read endpoints.
Note: Site Manager API (early access) doesn't appear to expose a
connected-client list directly. For endpoint discovery (IP + MAC of
connected clients like laptops, IoT, etc.) we'd still need to hit
each local controller's REST API — follow-up if the infrastructure-
level data isn't enough.
Requires: curl (present), jq (apt install jq).
Bug: FortiOS 7.x ana-gw replied to 'execute dhcp lease-list all' with
"Interface name 'all' does not exist." — my error-pattern grep didn't
include that phrase, so the script thought it got valid data, bailed
out of the retry loop, and handed empty/garbage to the parser, which
produced zero output with no error.
Fix: try the plain `execute dhcp lease-list` form first (works across
versions we've seen), fall back to the `all` variant only if the plain
form returns nothing. Validate acceptance by grepping for an actual
IP-shaped token — the parser needs IPs anyway, so "got real data"
and "has at least one IP" are equivalent conditions.
Device has been removed from the NH3 site. Drop the homepage card
and the corresponding example in discover-fortigate.sh.
Note left in services.yaml so whoever adds the replacement edge
device knows where the old entry lived.
First live run against ana-fw.phasefinal.com surfaced two bugs:
1. Script double-prefixed user@ when the arg already contained it
(e.g. `admin@10.250.250.1` became `admin@admin@10.250.250.1` →
auth prompt loop). Accept either "host" or "user@host" and only
prepend the default user if missing.
2. Parser assumed the wrong output format. Real FortiOS (tested on
7.x) emits:
<prompt> # <iface>
IP MAC-Address Hostname VCI SSID AP SERVER-ID Expiry
10.x.x.x ...
<next-iface>
IP MAC-Address ...
- Interface names are flush-left (no "Interface:" prefix)
- First line has the shell prompt embedded before the iface
- Hostnames don't contain spaces in practice
- 8 columns, not 4; VCI can contain "udhcp 1.32.1" etc.
Rewrote awk to detect interfaces via indentation (flush-left = iface,
indented = header or lease) and extract IP/MAC/Hostname from the
first three tokens of each lease row.
Verified against a captured sample; emits clean TSV.
First real run returned empty and we had no idea why — the script was
silently swallowing stderr via `2>/dev/null`. Remove the suppression
and try both `execute dhcp lease-list all` and the no-arg form, keeping
whichever returns non-error output.
Also emit a clearer diagnostic when both fail, pointing the user at
an interactive SSH to poke at command syntax.
Three scripts that surface hosts on the fleet's networks that aren't
already tracked under servers/*/. Goal: spot servers that need management
coverage (inventory, backup, monitoring) without wandering the LAN by
hand.
discover-fortigate.sh SSH to a FortiGate admin, run
`execute dhcp lease-list all`, emit TSV
(IP, MAC, hostname, vdom, source).
SSH was picked over the REST API for now
because it needs no API-token plumbing. The
parser is defensive about FortiOS output
format drift (multiple VDOM sections,
optional hostname). API variant can replace
it when the extra robustness is worth the
token setup.
discover-unifi.sh Cookie-auth REST call against a UniFi
Controller. Tries /api/auth/login (UniFi OS
/ UDM / Cloud Key Gen2+) first; falls back
to legacy /api/login for self-hosted
controllers. Output is the same TSV shape
as the FortiGate script so the two mix.
Needs UNIFI_USER / UNIFI_PASS env and jq.
discover-gaps.sh Consumes one or more TSVs from the sources
above. Builds the set of managed IPs from
servers/*/ssh-target (plus a grep of README
files for documented IPs) and prints any
discovered IPs not in that set.
Exit code is 1 if gaps found — suitable for
cron alerting.
Common pipeline:
scripts/discover-fortigate.sh ana-fw.phasefinal.com > /tmp/ana.tsv
scripts/discover-fortigate.sh nh3-gw.phasefinal.com > /tmp/nh3.tsv
UNIFI_USER=admin UNIFI_PASS=… scripts/discover-unifi.sh esh-uc.esteban.net > /tmp/esh.tsv
scripts/discover-gaps.sh /tmp/ana.tsv /tmp/nh3.tsv /tmp/esh.tsv
First-time use probably needs:
- SSH access configured to each FortiGate (admin login, key preferred)
- UniFi user with read access (the built-in API read-only role works)
- `jq` installed on the runner (for UniFi script)
nh3-dev is the author's active dev workstation at the NH3 site
(nh3-dev.phasefinal.com). 22 GB home with 12 GB of development code,
shell history, dotfiles, Claude Code memory, and scripts that weren't
being captured by any fleet backup.
Writes site-local to the Synology rest-server (10.100.50.50:8000)
matching the nh3-docker pattern.
Excludes trim ~8 GB of regenerable caches + build outputs:
- language toolchains (.cargo, .rustup, .npm, .m2, .gradle, go/pkg, etc.)
- editor caches (.vscode-server, .vscode)
- project build directories (node_modules, __pycache__, .venv, target,
dist, build, .pytest_cache, .tox, .next, .nuxt) via `**/` patterns
- trash / browser caches / steam / snap
Expected first snapshot ~14 GB, incrementals much smaller due to
content-defined dedup across dev code.
README walks through the full setup: resticprofile install, Synology-
side htpasswd addition (via DSM File Station or Container Manager since
the DSM SSH setup is tabled), repo init, profile deploy, timer enable,
first backup. Schedule is nightly 01:00 with systemd's Persistent=true
behavior so missed runs catch up at next boot (workstation is often
off/sleeping at backup time).
README.md fleet-coverage table updated to reflect 6/6 file-level
coverage (ana-docker, ana-ml2, nh3-docker, esh-docker-vm, vm-esh-nas,
nh3-dev).
Closes the last file-level backup gap. Primary ESH home-lab Docker host
— five services with state worth consistent dumps:
- paperless-ngx → external Postgres on 10.0.50.60 (host pg_dump)
- home-assistant → local SQLite ~50MB (host sqlite3 .backup)
- calibre-web-automated → local SQLite (in-container sqlite3)
- pgadmin → local SQLite (host sqlite3)
- uptime-kuma → local SQLite (host sqlite3; container name may vary
after force-recreate, detect by label)
Unique to this host: HA/pgadmin/uptime-kuma images don't bundle sqlite3.
Rather than maintaining custom images, pre-backup.sh runs sqlite3 from
the HOST against the volume bind-mount paths. Requires sqlite3 +
postgresql-client installed on esh-docker-vm.
Cross-site writes to rest-server-ana since ESH has no local rest-server.
NFS mounts (/mnt/{backup,books,compose,documents}) explicitly excluded
— hundreds of GB of NAS-side content backed up at the NAS layer. Also
excludes offen-sidecar buffer volumes (paperless + pgadmin currently
run offen/docker-volume-backup alongside — retire once restic has a
week of clean runs).
Found in audit (non-blocking but noted in README follow-ups):
- paperless-ngx Postgres password is literally "paperless-ng" —
trivially weak, rotate at next opportunity.
The Anaheim NAS exports /mnt/backup with root_squash, so sudo-as-root
on ana-docker becomes nobody on the NFS side and `sudo tee` gets EACCES.
The repo tree is owned by lkraven:lkraven precisely because of this —
all writes go through the lkraven UID.
Drop sudo from the example; plain `>>` append as lkraven works.
Second ESH Docker host — light (filezilla + dockge + agents, /opt/docker
is 56 KB). Cross-site writes to the Anaheim rest-server since ESH has
no local rest-server.
Critical detail: vm-esh-nas NFS-mounts /mnt/{share,music,books,media}
from 10.0.50.50 (~400 TB). Profile's exclude patterns explicitly
reject those paths as a safety net on top of the source list not
including them — a careless future edit to sources can't nuke the
backup repo by pulling in 400 TB.
Sources include /home/ (lkraven/nas/user dirs) in addition to the
usual /opt/docker /etc /root /var/lib/docker/volumes — this host has
multiple user accounts worth preserving dotfiles for.
No pre-backup hook: no relational DBs.
README walks through the full setup: install restic (not present on
this host), add htpasswd entry on rest-server-ana, install creds,
init, install resticprofile, schedule timers. Also flags the 3.8 GB
RAM constraint.
The compressed zip defeated restic's content-defined chunking: each
day's dump looked completely different to restic even when the repo
content barely changed, causing repo growth of ~full dump size (821 MB
at last measurement) every day until forget/prune aged snapshots out.
Uncompressed tar is dedup-friendly. After the first snapshot, daily
incrementals cost only the actual new-data delta — typically a few
MB for an active repo.
Tradeoff: stage file on the client host is ~2-3x the zip size while
the dump is in flight, but that's transient (purged at the start of
each run). Repo-side storage is much smaller over time.
Adds file-level restic for the NH3 Docker VM. Targets the Synology
rest-server (rest-server-nh3 at 10.100.50.50:8000) as user nh3-docker —
site-local writes matching the fleet pattern.
No pre-backup hook needed: none of the stacks on this host (adguard,
dockge, beszel-agent, dozzle-agent, portainer) run relational DBs. The
SQLite state in their named volumes is WAL-mode and restores cleanly
from raw restic capture.
Source paths mirror ana-ml2's template (/opt/docker, /etc, /root,
/var/lib/docker/volumes) with the same exclude list (docker internals,
logs, pids, root's shell/caches). ~1.6 MB of compose files + small
volumes — expected snapshot size a few hundred MB at most, dominated
by any AdGuard query log history.
README walks through reuse (existing repo + htpasswd) vs fresh init
paths, resticprofile install, timer generation, and verification via
the Backrest UI.
Audit surfaced three DB-backed services not being dumped consistently
by the existing pre-backup.sh:
- vaultwarden — migrated to external Postgres on PFI-Postgres
(10.250.50.80); old sqlite block was dumping stale pre-migration
files. Replace with pg_dump against the live database. Requires
postgresql-client on ana-docker and VW_PG* creds in
/etc/restic/dbcreds.env.
- gitea — also on PFI-Postgres; no hook existed at all. Use
`gitea dump` for a single zip that captures DB + repos + config +
LFS + attachments consistently. No explicit creds needed; the
container reads its own GITEA__database__* env.
- openwebui — two local SQLite databases (webui.db + the ChromaDB
vector store). .backup command if sqlite3 is in the image, volume-
level fallback otherwise.
Refactor: each block now logs a WARN on failure instead of aborting the
whole script — partial dumps > no dumps when one service has an issue.
dbcreds.env.example committed as a template; real file goes to
/etc/restic/dbcreds.env root:600 on the host and is never in the repo.
Mattermost retired (user confirmed 2026-04-21); removed from the
pre-backup.sh list and flagged in README's stacks section. Mattermost
container was not running regardless; the audit surfaced that it was
already effectively gone. Compose dir on ana-docker can be removed as
separate cleanup.
First backup run pulled in 9 GB due to /var/lib/docker/volumes/
parakeet_parakeet_cache — Parakeet is the only AI stack on ana-ml2
using a docker named volume for its HF model cache (kokoro, vibevoice,
comfyui, llama-swap, vllm-qwen3 all bind-mount from /tank which is
already outside source paths).
Excluding brings expected snapshot size back to ~100-300 MB.
rest-server-ana README now describes the /mnt/backup/restic/repo/
top-level NFS mount and its three per-site subdirs:
ana/ — live data served by this rest-server (per-host repos +
.htpasswd) — what DATA_DIR points at
esh/ — mirror destination for ESH-site backups (pending)
nh3/ — mirror destination for NH3 Synology's tree (pending)
ana-ml2 README gains a proper "Recreating the repo" section with the
correct /mnt/backup/restic/repo/ana/ana-ml2/ path for wiping the old
repo after a lost passphrase, and two paths for regenerating keys:
- interactive: type a user-generated passphrase at restic's init
prompt, then install it into /etc/restic/password via `cat > file`
+ Ctrl-D (no shell history or transcript exposure)
- scripted: openssl rand -base64 48, passphrase prints once and must
be captured into the password manager immediately
Cross-site replication snippet in rest-server-ana README updated to
use the unified /mnt/backup/restic/repo/{esh,nh3}/ destinations
instead of the earlier restic-mirror-*/ staging paths.
Original used `sudo env \$(cat /etc/restic/restic.env) …` but the `cat`
in command substitution runs as the login user, not root. Since the
env-file is root:600, the substitution silently yielded an empty
RESTIC_REPOSITORY and restic errored with "Please specify repository
location".
Wrap the whole dance in `sudo bash -c "…"` so the env-file read and
the restic call both happen as root.
An ana-ml2 user and repo were created during the original backup
pipeline pass. Reuse keeps snapshot history consolidated and avoids
duplicate infrastructure. Reworks the README to:
- Skip `restic init` (repo exists) and the htpasswd step (user exists)
- Install the two existing secrets (REST URL w/ htpasswd password,
repo passphrase) into /etc/restic/{restic.env,password}
- Verify credentials against the existing repo via `restic snapshots`
Fresh-setup flow retained below as a fallback for zero-state rebuilds.
ana-ml2 is not on any Proxmox hypervisor, so vzdump doesn't touch it.
This closes the biggest single backup gap per the 2026-04-20 pipeline
audit.
Sources: /opt/docker (~110 MB), /etc, /root, /var/lib/docker/volumes.
Excludes /tank/* (model weights — regenerable from Hugging Face and
would blow repo size budget). No pre-backup DB hook — none of the
llama-swap / vllm / comfyui / kokoro / parakeet / vibevoice stacks
use relational databases.
README walks through the one-time setup: rest-server .htpasswd entry,
restic init with fresh passphrase, resticprofile install, systemd timer
generation, verification against the Backrest UI.
Parallel to refresh-server-info.sh but pipes proxmox_inspect.sh and writes
to servers/<host>/proxmox-details.txt. Same discovery / ssh-target /
validate-only / dry-run behavior.
Fleet-wide `all` matches dir names containing `-pve` (covers *-pve and
*-pve-* so esh-pve-nas is included alongside pfi-pve / nh3-pve / esh-pve).
Explicit names are never filtered — useful for one-off PVE hosts with
non-matching names.
Validation checks the captured snapshot for: truncation, missing PVE
version line, non-Proxmox target, and surfaces backup-coverage "NO"
verdict counts so gaps show up in the validate-only output.
Initial fleet snapshot refreshed.
services.yaml:
- ANA-Firewall: href + siteMonitor both point at the IP now (was href
to FQDN but siteMonitor to IP — inconsistent)
- PFI-VM-Docker, NH3-SW1, NH3-VM-Docker, ESH-VM-Docker: upgrade from
`ping:` to `siteMonitor:` against the href URL so the up/down dot
reflects whether the web UI actually responds, not just ICMP
dockge canonical:
- icon sh-dockge.png (was si-portainer — wrong project). Applies to all
five fleet Dockge instances once their compose files are redeployed.
Per-group colors investigated but not supported by homepage (only site-
wide `color:` exists); skipped.
Flat-list layout didn't scale well once auto-discovery filled Apps and
Service Networking with a dozen+ cards each. Splits the dashboard:
Main — Monitoring, AI Systems, Apps, Media, Games, UltraSeedbox
Infra — three per-site hardware groups
Plumbing — Service Networking (dockge x5, traefik, adguard, etc.)
Row counts set to 4 columns on dense groups so they render as grids
rather than vertical walls.
Closes the last open backup-coverage gap identified in the 2026-04-20
audit. All guests on all four hypervisors are now covered by vzdump
jobs (pfi-pve 11/11, nh3-pve 5/5, esh-pve 3/3, esh-pve-nas 5/5).
VM images are backed up via Proxmox vzdump, but ana-ml2 (bare metal) has
no backup at all, and file-level restic + DB dumps are still missing on
three of four Docker-host VMs. Adds an explicit coverage table to the
README backup section so the gaps don't get overlooked while planning.
Goal (per user): every Docker host + configs + every database, not just
the VM image layer.
CWA image ships curl and busybox nc but no wget, so the initial
healthcheck wedged the container in unhealthy state. Swap to curl -fsS
hitting the login page; compose-time substitution of CWA_PORT keeps
custom port overrides working.
Read-only companion to server_inspect.sh. Runs over ssh via stdin the
same way:
ssh root@pfi-pve 'bash -s' < scripts/proxmox_inspect.sh \\
> servers/pfi-pve/proxmox-details.txt
Captures what the generic inspect misses on PVE nodes:
- Cluster + node resources (pvesh get /cluster/resources)
- VM list + per-vmid config highlights (qm list, qm config)
- LXC list + per-ctid config highlights (pct list, pct config)
- Storage pools (pvesm status, zpool, storage.cfg, NFS mounts)
- Backup jobs table parsed from /etc/pve/jobs.cfg (no jq needed)
- Per-guest backup coverage verdict: YES/NO with reason
- Most-recent actual backup per VMID on each local dump storage
- PBS targets, replication jobs, listening ports, subscription status
Coverage analysis surfaces the real gap: on pfi-pve, VMIDs 106, 107,
109, 112, 113 have no scheduled backup; VMID 106 (PFI-Tailscale) has
not been backed up since 2024-05-31. nh3-pve uses "all" job and covers
everything. esh-pve + esh-pve-nas share the tank-vmbu dump storage via
NFS; coverage splits cleanly across the two nodes.
First snapshots committed for each PVE host.
Retire the linuxserver.io calibre + calibre-web pair. CWA
(crocodilestick/calibre-web-automated) is an all-in-one: combined web UI
with Calibre backend, auto-ingest via drop folder, scheduled metadata
maintenance — single container on :8083.
- Reuses the existing library at /mnt/books/calibre/calibre_library
(NFS) and the existing calibre-web config at /opt/docker/conf/calibre-web,
so user accounts and the book catalogue carry over.
- Requires NETWORK_SHARE_MODE=true (library is on NFS; SQLite WAL would
corrupt otherwise).
- New drop-zone at /mnt/books/calibre/ingest; files placed there are
ingested into the library and then removed.
- Loses the LSIO desktop Calibre GUI (ports 8082/8181/8281) — run
Calibre locally if you need the desktop UI.
Old compose dirs (/opt/docker/compose/{calibre,calibre-web}) removed on
esh-docker-vm; old images can be pruned at leisure.
vm-esh-nas (10.0.50.154) is a NAS-adjacent Docker VM on the esh-pve-nas
hypervisor. Runs filezilla (port 5800), dockge, beszel-agent, dozzle-agent
with /mnt/{share,music,books,media} NFS-mounted from 10.0.50.50.
Use this host when a stack needs direct NFS mounts to the ESH NAS shares.
Canonicalize dockge as stacks/dockge/ — single compose used on all five
Docker hosts with per-host DOCKGE_HOST_LABEL/DOCKGE_HOST_IP in .env so
each card on the homepage points at its own instance. Labeled
homepage.group=Service Networking.
Beszel + Dozzle agent dirs also renamed to beszel-agent-<site> /
dozzle-agent-<site> pattern across the fleet for consistency.
Reorganize the gethomepage dashboard from site-based (PFI-ANA, ESH, NH3)
to function-first grouping (Monitoring, AI Systems, Apps, Media, Games,
Infra-<site>, Service Networking). Canonical config now tracked in
configs/homepage/ with Plex/Jellyfin widget keys moved to env
substitution.
Label sweep across fleet compose files:
- beszel, dozzle, backrest -> Monitoring
- rest-server-ana -> Service Networking
Healthcheck fixes (previous wget/curl paths broke on distroless +
--private-repos 401):
- beszel hub: /beszel health --url ...
- beszel agent: /agent health (newly added)
- rest-server: nc -z localhost 8000 (TCP probe)
Group name originally "Wiring / Plumbing" collapsed to single-word group
on homepage's parser; renamed to "Service Networking" everywhere.
pfi-pve (Anaheim), nh3-pve (NH), esh-pve + esh-pve-nas (ESH home lab) now
have servers/<host>/ entries with ssh-target files, README stubs, and
first snapshots. Register all four via add-host.sh + ssh-copy-id; SSH
auth is root@<name> with key-based access.
Fix a resolver bug in the three workspace scripts: previously, when a
hostname resolved via /etc/hosts, resolve_target returned the bare name
and ssh dropped the user prefix stored in ssh-target (connecting as the
wrong user). Now ssh-target wins whenever it's present, since it can
carry user@ or port info that /etc/hosts + ssh_config cannot. Falls back
to the dir name only when no ssh-target file exists.
CLAUDE.md and README.md split the fleet table into Docker hosts vs
hypervisors with an explicit placement rule: don't deploy Docker stacks
on PVE hosts; capture host-level detail with server_inspect.sh, use
Proxmox-native tooling (qm list, pvesh, zpool) for VM/storage audits.
Network probes (no web admin, Debian SSH banner, only file-server ports
open) show 10.250.50.50 is vanilla Debian 12 with hand-configured NFS/SMB,
reprovisioned from the original TrueNAS SCALE install. Update stack
comments, README storage notes, Backrest description, proxmox-vms.md
entry for VM 100, and the restic configs intro to match.
Cross-site sync plan simplifies to plain rsync over SSH on both sides —
no appliance-specific tooling needed.
Upstream compose files on both hosts now reference ${DB_ROOT_PW} (seafile)
and ${ADGUARD_WIDGET_PASSWORD} (esh adguard) from their respective .env,
so next refreshes no longer capture the literal passwords.
Captures the full workspace state built up to this point:
- CLAUDE.md + README.md describing conventions and the four-host fleet
(ana-ml2, ana-docker, nh3-docker, esh-docker-vm).
- Per-host notes under servers/<host>/ with ssh-target fallback files
and latest system-details snapshots (two in-compose credential leaks
scrubbed; the upstream compose files still need to move those to .env).
- scripts/: server_inspect.sh (read-only remote diagnostic),
refresh-server-info.sh (dir-driven discovery + snapshot capture with
validation warnings), add-host.sh, sync-stacks.sh (pull
compose/conf trees), deploy-stack.sh (push with per-file diff + prompt).
- stacks/: canonical compose for backrest, beszel, dozzle, llama-swap,
rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired
infinity reference. All use the .env-driven + traefik-net + homepage
label pattern.
- configs/restic/ana-docker/: first resticprofile config + pre-backup
hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates
for the other three hosts to come.
- docs/pfi/: general infrastructure reference carried over.
- .gitignore excludes .env, stacks-mirror/, and assorted secret/state
filenames to prevent re-leaks on later commits.
vh