CWA image ships curl and busybox nc but no wget, so the initial
healthcheck wedged the container in unhealthy state. Swap to curl -fsS
hitting the login page; compose-time substitution of CWA_PORT keeps
custom port overrides working.
Read-only companion to server_inspect.sh. Runs over ssh via stdin the
same way:
ssh root@pfi-pve 'bash -s' < scripts/proxmox_inspect.sh \\
> servers/pfi-pve/proxmox-details.txt
Captures what the generic inspect misses on PVE nodes:
- Cluster + node resources (pvesh get /cluster/resources)
- VM list + per-vmid config highlights (qm list, qm config)
- LXC list + per-ctid config highlights (pct list, pct config)
- Storage pools (pvesm status, zpool, storage.cfg, NFS mounts)
- Backup jobs table parsed from /etc/pve/jobs.cfg (no jq needed)
- Per-guest backup coverage verdict: YES/NO with reason
- Most-recent actual backup per VMID on each local dump storage
- PBS targets, replication jobs, listening ports, subscription status
Coverage analysis surfaces the real gap: on pfi-pve, VMIDs 106, 107,
109, 112, 113 have no scheduled backup; VMID 106 (PFI-Tailscale) has
not been backed up since 2024-05-31. nh3-pve uses "all" job and covers
everything. esh-pve + esh-pve-nas share the tank-vmbu dump storage via
NFS; coverage splits cleanly across the two nodes.
First snapshots committed for each PVE host.
Retire the linuxserver.io calibre + calibre-web pair. CWA
(crocodilestick/calibre-web-automated) is an all-in-one: combined web UI
with Calibre backend, auto-ingest via drop folder, scheduled metadata
maintenance — single container on :8083.
- Reuses the existing library at /mnt/books/calibre/calibre_library
(NFS) and the existing calibre-web config at /opt/docker/conf/calibre-web,
so user accounts and the book catalogue carry over.
- Requires NETWORK_SHARE_MODE=true (library is on NFS; SQLite WAL would
corrupt otherwise).
- New drop-zone at /mnt/books/calibre/ingest; files placed there are
ingested into the library and then removed.
- Loses the LSIO desktop Calibre GUI (ports 8082/8181/8281) — run
Calibre locally if you need the desktop UI.
Old compose dirs (/opt/docker/compose/{calibre,calibre-web}) removed on
esh-docker-vm; old images can be pruned at leisure.
vm-esh-nas (10.0.50.154) is a NAS-adjacent Docker VM on the esh-pve-nas
hypervisor. Runs filezilla (port 5800), dockge, beszel-agent, dozzle-agent
with /mnt/{share,music,books,media} NFS-mounted from 10.0.50.50.
Use this host when a stack needs direct NFS mounts to the ESH NAS shares.
Canonicalize dockge as stacks/dockge/ — single compose used on all five
Docker hosts with per-host DOCKGE_HOST_LABEL/DOCKGE_HOST_IP in .env so
each card on the homepage points at its own instance. Labeled
homepage.group=Service Networking.
Beszel + Dozzle agent dirs also renamed to beszel-agent-<site> /
dozzle-agent-<site> pattern across the fleet for consistency.
Reorganize the gethomepage dashboard from site-based (PFI-ANA, ESH, NH3)
to function-first grouping (Monitoring, AI Systems, Apps, Media, Games,
Infra-<site>, Service Networking). Canonical config now tracked in
configs/homepage/ with Plex/Jellyfin widget keys moved to env
substitution.
Label sweep across fleet compose files:
- beszel, dozzle, backrest -> Monitoring
- rest-server-ana -> Service Networking
Healthcheck fixes (previous wget/curl paths broke on distroless +
--private-repos 401):
- beszel hub: /beszel health --url ...
- beszel agent: /agent health (newly added)
- rest-server: nc -z localhost 8000 (TCP probe)
Group name originally "Wiring / Plumbing" collapsed to single-word group
on homepage's parser; renamed to "Service Networking" everywhere.
pfi-pve (Anaheim), nh3-pve (NH), esh-pve + esh-pve-nas (ESH home lab) now
have servers/<host>/ entries with ssh-target files, README stubs, and
first snapshots. Register all four via add-host.sh + ssh-copy-id; SSH
auth is root@<name> with key-based access.
Fix a resolver bug in the three workspace scripts: previously, when a
hostname resolved via /etc/hosts, resolve_target returned the bare name
and ssh dropped the user prefix stored in ssh-target (connecting as the
wrong user). Now ssh-target wins whenever it's present, since it can
carry user@ or port info that /etc/hosts + ssh_config cannot. Falls back
to the dir name only when no ssh-target file exists.
CLAUDE.md and README.md split the fleet table into Docker hosts vs
hypervisors with an explicit placement rule: don't deploy Docker stacks
on PVE hosts; capture host-level detail with server_inspect.sh, use
Proxmox-native tooling (qm list, pvesh, zpool) for VM/storage audits.
Network probes (no web admin, Debian SSH banner, only file-server ports
open) show 10.250.50.50 is vanilla Debian 12 with hand-configured NFS/SMB,
reprovisioned from the original TrueNAS SCALE install. Update stack
comments, README storage notes, Backrest description, proxmox-vms.md
entry for VM 100, and the restic configs intro to match.
Cross-site sync plan simplifies to plain rsync over SSH on both sides —
no appliance-specific tooling needed.
Upstream compose files on both hosts now reference ${DB_ROOT_PW} (seafile)
and ${ADGUARD_WIDGET_PASSWORD} (esh adguard) from their respective .env,
so next refreshes no longer capture the literal passwords.
Captures the full workspace state built up to this point:
- CLAUDE.md + README.md describing conventions and the four-host fleet
(ana-ml2, ana-docker, nh3-docker, esh-docker-vm).
- Per-host notes under servers/<host>/ with ssh-target fallback files
and latest system-details snapshots (two in-compose credential leaks
scrubbed; the upstream compose files still need to move those to .env).
- scripts/: server_inspect.sh (read-only remote diagnostic),
refresh-server-info.sh (dir-driven discovery + snapshot capture with
validation warnings), add-host.sh, sync-stacks.sh (pull
compose/conf trees), deploy-stack.sh (push with per-file diff + prompt).
- stacks/: canonical compose for backrest, beszel, dozzle, llama-swap,
rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired
infinity reference. All use the .env-driven + traefik-net + homepage
label pattern.
- configs/restic/ana-docker/: first resticprofile config + pre-backup
hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates
for the other three hosts to come.
- docs/pfi/: general infrastructure reference carried over.
- .gitignore excludes .env, stacks-mirror/, and assorted secret/state
filenames to prevent re-leaks on later commits.
vh