Bundles the inventory expansion since 2026-04-22:
- New host dirs (READMEs + ssh-target where dir name doesn't resolve):
ana-nas, ana-wg, esh-vm-db, nh3-nas, pbs-ana, pbs-nh3.
- New PFI VM snapshots (registered + key-installed 2026-04-23):
ana-filebot, pfi-ana-webhost, pfi-postgres, pfi-pteradactyl,
pfi-tacticalrmm, sf-ana-container, sfsrv-ana (system + proxmox).
- servers/irv-ml1: ONBOARDING.md (the first-time setup notes from when
the host was brought into the fleet) + ssh-target (10.100.79.3 over
the WG tunnel — name doesn't DNS-resolve from this workstation).
- servers/{ana-ml2,pfi-pve,sf-r630}/README.md: updates to capture BMC
IPs, the iDRAC vs OS hostname distinction (sf-r630 hardware =
sfsrv-ana OS), and the ana-ml2 Supermicro BMC (10.250.250.50,
distinct from the Dell R750xs iDRAC).
- configs/homepage/docker.yaml: irv-ml1-docker provider added so
homepage auto-discovers irv-ml1's stacks over the WG tunnel.
- docs/orientation.md: narrative fleet overview written for fresh
Claude sessions — sites, backup architecture, governing principles,
gotchas, where-to-look guide. Pointed at from CLAUDE.md.
New stack mirroring PFI convention (stacks/comfyui/) using
mmartial/comfyui-nvidia-docker:ubuntu24_cuda12.8-20260312. Both GPUs
exposed, pinned to CUDA 12.8 to match the host's 570.x driver and the
native cuda-toolkit already in place.
Layout — single tree under /worktank/comfyui/ (462G dedicated, 1%
used pre-deploy):
- basedir/ → /basedir user state (models, workflows, custom_nodes,
input, output); owned 1000:1000 so external
tools can edit workflow JSON directly.
- run/ → /comfy/mnt ComfyUI source + venv + pip cache (~7.8G
after bootstrap). Bind mount instead of
named volume — the image refuses to chown
mounted paths at startup, so keeping this
lkraven-owned avoids the sudo dance.
servers/irv-ml1/README.md refreshed: Docker upgraded to 29.4.1 with
traefik-net in place; dockge + beszel-agent + dozzle-agent already
present; /storetank dropped 92% → 64%; restic coverage to
rest-server-nh3 is operational (not "currently none" as prior text).
Initial framing was wrong. PFI runs these under a managed-hosting
agreement: SSH, OS ops, backups are all PFI's responsibility. Hardware
and data belong to the client.
Changes:
- ssh-target files added for sfsrv-ana (root@10.250.250.115 — same
pattern as other PVE nodes) and sf-ana-container
(lkraven@10.250.150.100 guess, adjust if different user).
- sf-r630 still lacks an ssh-target — the OS-side LAN IP isn't in
FortiGate DHCP (static config somewhere). Will fill in once
identified; README flags that gap.
- READMEs rewritten: dropped "tenant-scoped" / "not SSH-managed"
language, added "client context" section that explains the
managed-hosting relationship. Backup coverage now listed as
planned rather than blocked on tenant coordination.
- CLAUDE.md fleet table: SF rows re-labeled "SureFire client
(PFI-managed)". Placement-rules section updated to note that
SF hosts are first-class PFI-ops targets, just client-owned.
- Memory (project_surefire_tenant.md) rewritten to reflect
managed-services reality + hosts-file entries needed for name
resolution since these aren't in PFI DNS.
Six PFI VMs/LXCs previously known only via proxmox_inspect.sh —
covered by vzdump but not in servers/, so operational context
(roles, backup posture, ssh target) was missing:
pfi-ana-webhost (VMID 110) — web workload
ana-filebot (LXC 112) — file-task automation
pfi-pteradactyl (VMID 107) — Pterodactyl game panel
pfi-tacticalrmm (VMID 111) — TacticalRMM remote-management
pfi-postgres (VMID 105) — shared Postgres (vaultwarden/gitea/
paperless backends)
ana-wg (LXC 113) — WireGuard VPN gateway
Plus three SureFire tenant hosts at the Anaheim colo:
sfsrv-ana — tenant Proxmox hypervisor (10.250.250.115:8006)
sf-ana-container — container workload on that Proxmox
sf-r630 — physical R630 (iDRAC 10.250.250.110 for PFI-side
hardware mgmt; OS is tenant-scoped)
Each server dir has README + ssh-target where applicable. SureFire
entries explicitly document tenancy scope: PFI provides hosting,
SureFire owns the OS; management actions need tenant coordination.
SureFire hosts have no ssh-target by default.
Homepage Infra - ANA gains two new cards:
- SFsrv-ANA (https://10.250.250.115:8006, si-proxmox icon)
- SF-R630-iDRAC (https://10.250.250.110, si-dell icon)
PFI-ANA-ML2 BMC gained an href since it has a usable web UI.
CLAUDE.md fleet table extended with all 9 new rows. Placement-rules
section notes the SureFire tenant boundary.
Memory: new project_surefire_tenant.md so future sessions know sf-*
hosts are tenant-scoped by default.
Parallel to refresh-server-info.sh but pipes proxmox_inspect.sh and writes
to servers/<host>/proxmox-details.txt. Same discovery / ssh-target /
validate-only / dry-run behavior.
Fleet-wide `all` matches dir names containing `-pve` (covers *-pve and
*-pve-* so esh-pve-nas is included alongside pfi-pve / nh3-pve / esh-pve).
Explicit names are never filtered — useful for one-off PVE hosts with
non-matching names.
Validation checks the captured snapshot for: truncation, missing PVE
version line, non-Proxmox target, and surfaces backup-coverage "NO"
verdict counts so gaps show up in the validate-only output.
Initial fleet snapshot refreshed.
Closes the last open backup-coverage gap identified in the 2026-04-20
audit. All guests on all four hypervisors are now covered by vzdump
jobs (pfi-pve 11/11, nh3-pve 5/5, esh-pve 3/3, esh-pve-nas 5/5).
Read-only companion to server_inspect.sh. Runs over ssh via stdin the
same way:
ssh root@pfi-pve 'bash -s' < scripts/proxmox_inspect.sh \\
> servers/pfi-pve/proxmox-details.txt
Captures what the generic inspect misses on PVE nodes:
- Cluster + node resources (pvesh get /cluster/resources)
- VM list + per-vmid config highlights (qm list, qm config)
- LXC list + per-ctid config highlights (pct list, pct config)
- Storage pools (pvesm status, zpool, storage.cfg, NFS mounts)
- Backup jobs table parsed from /etc/pve/jobs.cfg (no jq needed)
- Per-guest backup coverage verdict: YES/NO with reason
- Most-recent actual backup per VMID on each local dump storage
- PBS targets, replication jobs, listening ports, subscription status
Coverage analysis surfaces the real gap: on pfi-pve, VMIDs 106, 107,
109, 112, 113 have no scheduled backup; VMID 106 (PFI-Tailscale) has
not been backed up since 2024-05-31. nh3-pve uses "all" job and covers
everything. esh-pve + esh-pve-nas share the tank-vmbu dump storage via
NFS; coverage splits cleanly across the two nodes.
First snapshots committed for each PVE host.
Retire the linuxserver.io calibre + calibre-web pair. CWA
(crocodilestick/calibre-web-automated) is an all-in-one: combined web UI
with Calibre backend, auto-ingest via drop folder, scheduled metadata
maintenance — single container on :8083.
- Reuses the existing library at /mnt/books/calibre/calibre_library
(NFS) and the existing calibre-web config at /opt/docker/conf/calibre-web,
so user accounts and the book catalogue carry over.
- Requires NETWORK_SHARE_MODE=true (library is on NFS; SQLite WAL would
corrupt otherwise).
- New drop-zone at /mnt/books/calibre/ingest; files placed there are
ingested into the library and then removed.
- Loses the LSIO desktop Calibre GUI (ports 8082/8181/8281) — run
Calibre locally if you need the desktop UI.
Old compose dirs (/opt/docker/compose/{calibre,calibre-web}) removed on
esh-docker-vm; old images can be pruned at leisure.
vm-esh-nas (10.0.50.154) is a NAS-adjacent Docker VM on the esh-pve-nas
hypervisor. Runs filezilla (port 5800), dockge, beszel-agent, dozzle-agent
with /mnt/{share,music,books,media} NFS-mounted from 10.0.50.50.
Use this host when a stack needs direct NFS mounts to the ESH NAS shares.
Canonicalize dockge as stacks/dockge/ — single compose used on all five
Docker hosts with per-host DOCKGE_HOST_LABEL/DOCKGE_HOST_IP in .env so
each card on the homepage points at its own instance. Labeled
homepage.group=Service Networking.
Beszel + Dozzle agent dirs also renamed to beszel-agent-<site> /
dozzle-agent-<site> pattern across the fleet for consistency.
pfi-pve (Anaheim), nh3-pve (NH), esh-pve + esh-pve-nas (ESH home lab) now
have servers/<host>/ entries with ssh-target files, README stubs, and
first snapshots. Register all four via add-host.sh + ssh-copy-id; SSH
auth is root@<name> with key-based access.
Fix a resolver bug in the three workspace scripts: previously, when a
hostname resolved via /etc/hosts, resolve_target returned the bare name
and ssh dropped the user prefix stored in ssh-target (connecting as the
wrong user). Now ssh-target wins whenever it's present, since it can
carry user@ or port info that /etc/hosts + ssh_config cannot. Falls back
to the dir name only when no ssh-target file exists.
CLAUDE.md and README.md split the fleet table into Docker hosts vs
hypervisors with an explicit placement rule: don't deploy Docker stacks
on PVE hosts; capture host-level detail with server_inspect.sh, use
Proxmox-native tooling (qm list, pvesh, zpool) for VM/storage audits.
Network probes (no web admin, Debian SSH banner, only file-server ports
open) show 10.250.50.50 is vanilla Debian 12 with hand-configured NFS/SMB,
reprovisioned from the original TrueNAS SCALE install. Update stack
comments, README storage notes, Backrest description, proxmox-vms.md
entry for VM 100, and the restic configs intro to match.
Cross-site sync plan simplifies to plain rsync over SSH on both sides —
no appliance-specific tooling needed.
Upstream compose files on both hosts now reference ${DB_ROOT_PW} (seafile)
and ${ADGUARD_WIDGET_PASSWORD} (esh adguard) from their respective .env,
so next refreshes no longer capture the literal passwords.
Captures the full workspace state built up to this point:
- CLAUDE.md + README.md describing conventions and the four-host fleet
(ana-ml2, ana-docker, nh3-docker, esh-docker-vm).
- Per-host notes under servers/<host>/ with ssh-target fallback files
and latest system-details snapshots (two in-compose credential leaks
scrubbed; the upstream compose files still need to move those to .env).
- scripts/: server_inspect.sh (read-only remote diagnostic),
refresh-server-info.sh (dir-driven discovery + snapshot capture with
validation warnings), add-host.sh, sync-stacks.sh (pull
compose/conf trees), deploy-stack.sh (push with per-file diff + prompt).
- stacks/: canonical compose for backrest, beszel, dozzle, llama-swap,
rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired
infinity reference. All use the .env-driven + traefik-net + homepage
label pattern.
- configs/restic/ana-docker/: first resticprofile config + pre-backup
hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates
for the other three hosts to come.
- docs/pfi/: general infrastructure reference carried over.
- .gitignore excludes .env, stacks-mirror/, and assorted secret/state
filenames to prevent re-leaks on later commits.
vh