servers: add new host dirs, refresh fleet snapshots, orientation doc

Bundles the inventory expansion since 2026-04-22: - New host dirs (READMEs + ssh-target where dir name doesn't resolve): ana-nas, ana-wg, esh-vm-db, nh3-nas, pbs-ana, pbs-nh3. - New PFI VM snapshots (registered + key-installed 2026-04-23): ana-filebot, pfi-ana-webhost, pfi-postgres, pfi-pteradactyl, pfi-tacticalrmm, sf-ana-container, sfsrv-ana (system + proxmox). - servers/irv-ml1: ONBOARDING.md (the first-time setup notes from when the host was brought into the fleet) + ssh-target (10.100.79.3 over the WG tunnel — name doesn't DNS-resolve from this workstation). - servers/{ana-ml2,pfi-pve,sf-r630}/README.md: updates to capture BMC IPs, the iDRAC vs OS hostname distinction (sf-r630 hardware = sfsrv-ana OS), and the ana-ml2 Supermicro BMC (10.250.250.50, distinct from the Dell R750xs iDRAC). - configs/homepage/docker.yaml: irv-ml1-docker provider added so homepage auto-discovers irv-ml1's stacks over the WG tunnel. - docs/orientation.md: narrative fleet overview written for fresh Claude sessions — sites, backup architecture, governing principles, gotchas, where-to-look guide. Pointed at from CLAUDE.md.
2026-04-24 21:56:46 -07:00
parent 574c72daa5
commit 60367b307f
31 changed files with 2391 additions and 26 deletions
@@ -0,0 +1,77 @@
+# ana-nas
+
+Debian NFS/SMB file server at the Anaheim site — **CT 109** (LXC) on
+pfi-pve. Serves the fleet's primary storage layer:
+
+- `/mnt/backup` → restic target for `rest-server-ana` on ana-docker
+  + backing storage for PBS-ANA's datastore
+- `/mnt/docker` → compose/config shares
+- `/mnt/webdav` → WebDAV data (if still active)
+- Historically `/mnt/db` (retired 2026-04-23 after the DB-off-NFS
+  migration; pfi-postgres now uses local disk)
+
+## Network
+
+- **LAN IP:** 10.250.50.50
+- **SSH:** `ssh ana-nas` (config alias → `lkraven@10.250.50.50`,
+  key auth). lkraven is a standard user — sudo for root operations.
+
+## Container context
+
+This is **CT 109**, not a VM. Bind-mounts into the LXC from pfi-pve's
+`ospool` ZFS datasets:
+
+| Inside CT | Host path on pfi-pve | Purpose |
+|---|---|---|
+| `/mnt/ospool-backup` | ZFS dataset | ospool snapshots staging |
+| `/mnt/backup` | ZFS dataset | rest-server-ana + PBS-ANA data |
+| `/mnt/db` | ZFS dataset | **retired 2026-04-23** (postgres moved to local disk) |
+| `/mnt/docker` | ZFS dataset | compose files + conf shares |
+| `/mnt/pve-VMStorage` | ZFS dataset | alt VM storage pool |
+| `/mnt/webdav` | ZFS dataset | WebDAV data |
+
+Because these are bind-mounts, CT 109's own vzdump backup (LXC rootfs
+only) does NOT capture the data payload — the data lives on pfi-pve's
+ospool and is captured via the host's own backup chain.
+
+## Why CT 109 matters (SPOF warning)
+
+As of 2026-04-23, ana-nas still affects these fleet services if it
+goes offline:
+
+- **rest-server-ana** (on ana-docker) — its data dir is NFS-mounted
+  from here; an outage kills file-level backups fleet-wide
+- **PBS-ANA datastore** — NFS-mounted from here; an outage kills
+  VM-level backups fleet-wide
+- **Cross-site restic rsync source** (ana-nas → nh3-nas daily)
+
+**Not** affected anymore (as of 2026-04-23 DB migration):
+
+- pfi-postgres (VM 105) — now on local disk, ana-nas outages
+  don't cascade into vaultwarden/gitea/paperless/zammad
+
+Full recovery procedure: `memory/incident_ana_nas_spof.md` +
+`docs/runbooks/disaster-recovery.md`.
+
+## DSM / rescue access
+
+Under the hood this is vanilla Debian 12 with hand-configured NFS/SMB.
+No appliance tooling — share tweaks go through `/etc/exports` +
+`systemctl reload nfs-server`. Historical confusion about this box
+being a TrueNAS appliance persists in naming (the Proxmox label was
+`PFI-ANA-TRUENAS` for a long time) but it is plain Debian.
+
+## History note
+
+ID-over-time: this host has been called VM 100, PFI-ANA-TRUENAS,
+and now CT 109 / ana-nas. Memory file `storage_ana_nas.md` records
+the VM→CT correction that was clarified 2026-04-23 from the vzdump
+log output.
+
+## Refresh
+
+```
+scripts/refresh-server-info.sh ana-nas
+```
+
+Snapshot at `system-details.txt`.
@@ -0,0 +1 @@
+lkraven@10.250.50.50
@@ -0,0 +1,96 @@
+
+===== HOST =====
+
+Hostname:   ana-nas
+Date:       2026-04-23T06:53:27+00:00
+Uptime:     up 34 weeks, 3 days, 11 hours, 22 minutes
+OS:         Debian GNU/Linux 12 (bookworm)
+Kernel:     6.8.12-8-pve
+Arch:       x86_64
+
+===== HARDWARE =====
+
+CPU cores:  4
+CPU model:  Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz
+MemTotal:   2.0 GB
+MemAvailable: 1.9 GB
+
+===== GPUS =====
+
+nvidia-smi not present (no NVIDIA GPUs or driver not installed)
+
+===== FILESYSTEMS (df) =====
+
+Filesystem                Size  Used Avail Use% Mounted on
+ospool/subvol-109-disk-0   80G  813M   80G   1% /
+ospool/backup              11T  240M   11T   1% /mnt/ospool-backup
+NASPool/backupStore        20T   67G   20T   1% /mnt/backup
+NASPool/db                 20T  282M   20T   1% /mnt/db
+NASPool/docker             20T  384K   20T   1% /mnt/docker
+NASPool/pve-VMStorage      22T  2.4T   20T  12% /mnt/pve-VMStorage
+NASPool/webdav             20T  256K   20T   1% /mnt/webdav
+
+===== PERSISTENT MOUNTS (/etc/fstab, non-comment) =====
+
+
+===== TARGETED DATA PATHS =====
+
+/opt  (total: 512)
+  total 9
+  drwxr-xr-x  2 root root  2 2023-10-10 13:26 .
+  drwxr-xr-x 17 root root 21 2025-08-24 19:31 ..
+
+/srv  (total: 512)
+  total 9
+  drwxr-xr-x  2 root root  2 2023-10-10 13:26 .
+  drwxr-xr-x 17 root root 21 2025-08-24 19:31 ..
+
+
+===== DOCKER =====
+
+docker not installed
+
+===== COMPOSE FILES (/opt/docker/compose/) =====
+
+/opt/docker/compose not present
+
+===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) =====
+
+/opt/docker/conf not present
+
+===== LISTENING PORTS =====
+
+*:22
+*:9090
+0.0.0.0:111
+0.0.0.0:139
+0.0.0.0:2049
+0.0.0.0:39347
+0.0.0.0:40609
+0.0.0.0:445
+0.0.0.0:48585
+0.0.0.0:49363
+0.0.0.0:57311
+127.0.0.1:25
+[::1]:25
+[::]:111
+[::]:139
+[::]:2049
+[::]:41319
+[::]:445
+[::]:47023
+[::]:50729
+[::]:50951
+[::]:55887
+
+===== MODEL / HUGGINGFACE CACHES =====
+
+
+===== DOCKER-ADJACENT SYSTEMD SERVICES =====
+
+container-getty@1.service running
+container-getty@2.service running
+
+===== DONE =====
+
+Paste the above back into the chat, or pass a path as argv[1] to save.