servers: add new host dirs, refresh fleet snapshots, orientation doc

Bundles the inventory expansion since 2026-04-22: - New host dirs (READMEs + ssh-target where dir name doesn't resolve): ana-nas, ana-wg, esh-vm-db, nh3-nas, pbs-ana, pbs-nh3. - New PFI VM snapshots (registered + key-installed 2026-04-23): ana-filebot, pfi-ana-webhost, pfi-postgres, pfi-pteradactyl, pfi-tacticalrmm, sf-ana-container, sfsrv-ana (system + proxmox). - servers/irv-ml1: ONBOARDING.md (the first-time setup notes from when the host was brought into the fleet) + ssh-target (10.100.79.3 over the WG tunnel — name doesn't DNS-resolve from this workstation). - servers/{ana-ml2,pfi-pve,sf-r630}/README.md: updates to capture BMC IPs, the iDRAC vs OS hostname distinction (sf-r630 hardware = sfsrv-ana OS), and the ana-ml2 Supermicro BMC (10.250.250.50, distinct from the Dell R750xs iDRAC). - configs/homepage/docker.yaml: irv-ml1-docker provider added so homepage auto-discovers irv-ml1's stacks over the WG tunnel. - docs/orientation.md: narrative fleet overview written for fresh Claude sessions — sites, backup architecture, governing principles, gotchas, where-to-look guide. Pointed at from CLAUDE.md.
2026-04-24 21:56:46 -07:00
parent 574c72daa5
commit 60367b307f
31 changed files with 2391 additions and 26 deletions
@@ -21,6 +21,13 @@ ana-ml2-docker:
  host: 10.250.50.54
  port: 2375
 # irv-ml1 — reachable ONLY via WireGuard tunnel, so docker auto-discovery
 # traffic traverses the tunnel. If WG drops, homepage will show irv-ml1
 # services as offline until reconnect.
 irv-ml1-docker:
  host: 10.100.79.3
  port: 2375
 # Example TLS socket (if/when a host moves off plaintext 2375):
 # ana-pfi-docker:
 #   host: 10.250.50.70
@@ -0,0 +1,258 @@
 # Fleet orientation — start here
 Read this first if you're a fresh session joining this project. It
 summarizes the fleet, the conventions, the known gotchas, and points
 at the deeper docs / memory files that cover each area in detail.
 ## What this repo is
 A **reference + management workspace** for PFI infrastructure — a
 small multi-site fleet of Proxmox hypervisors, Linux VMs/LXCs, a
 bare-metal GPU host, and two NAS boxes. Authoritative copies of
 Docker compose files, backup configs, and homepage dashboard config
 live **on the hosts themselves**; this repo mirrors them for version
 control, planning, and cross-host orchestration.
 Primary file you'll read automatically on session start:
 - `CLAUDE.md` — conventions, servers table, scripts tour, placement rules.
 - `~/.claude/projects/.../memory/MEMORY.md` + linked memory files —
  durable cross-session facts.
 This file (`docs/orientation.md`) is the narrative overview that ties
 them together.
 ## Sites + hosts at a glance
 Three physical locations plus one WireGuard-tunneled satellite:
 | Site | Subnet | Physical boxes | Notes |
 |---|---|---|---|
 | **Anaheim (ANA)** | `10.250.0.0/16` | Dell R750xs (pfi-pve hypervisor, iDRAC 10.250.250.30), Supermicro (ana-ml2 bare-metal GPU, BMC 10.250.250.50), Dell R630 (sfsrv-ana SureFire tenant, iDRAC 10.250.250.110) | Primary production site |
 | **NH3** | `10.100.0.0/16` | nh3-pve hypervisor, Synology RS2418+ (`nh3-nas`, 42 TB) | Secondary + DR target |
 | **ESH home lab** | `esteban.net`, `10.0.50.0/24` | esh-pve, esh-pve-nas (split-role hypervisors), esh-vm-db (DB VM) | Non-production, shares backup tooling |
 | **Irvine (IRV)** | `10.100.79.0/24` (WG tunnel only) | irv-ml1 bare-metal GPU box | Reachable ONLY via WireGuard tunnel from NH3 |
 Full host list with roles: see the **Servers table** in `CLAUDE.md`.
 ## Backup architecture (two layers)
 This is the most fleet-impacting system. Every host gets backed up
 via one or both layers.
 ### Layer 1 — PBS (VM-image, block-level dedupe)
 - **Primary:** `pbs-ana` (VM on pfi-pve, NFS datastore on ana-nas).
 - **DR mirror:** `pbs-nh3` (VM on nh3-pve, NFS datastore on nh3-nas Synology).
 - **Sync:** pbs-nh3 pulls from pbs-ana at 06:00 daily. One-way.
 - **Namespaces:** per-hypervisor (`pfi-pve`, `nh3-pve`, `esh-pve`,
  `esh-pve-nas`, `sfsrv-ana`) because VMIDs are NOT globally unique.
 - **Scope:** all 5 hypervisors back up to pbs-ana nightly.
 Deployment runbook: `docs/runbooks/pbs-deployment.md` (9 phases,
 captured during the 2026-04-22 deployment).
 ### Layer 2 — restic (file-level, encrypted, DB-consistent)
 - **Target A:** `rest-server-ana` container on ana-docker. Data dir
  NFS-mounted from ana-nas. Used by: ana-docker, ana-ml2,
  esh-docker-vm, vm-esh-nas, esh-vm-db.
 - **Target B:** `rest-server-nh3` container on nh3-nas (Synology).
  Used by: nh3-docker, nh3-dev, irv-ml1.
 - **Mode:** both run `--append-only --private-repos` — clients can't
  delete their own history, and each client's URL path must match
  their HTTP basic-auth user.
 - **Scope:** 8 hosts have resticprofile + systemd timers firing at
  01:00 daily.
 - **DB-consistent dumps:** hosts with databases have a
  `pre-backup.sh` that runs pg_dump/mongodump/sqlite3 .backup into
  `/var/lib/restic/stage/` before restic sweeps.
 ### Cross-site restic mirror (rsync)
 - **ana-nas → nh3-nas** daily at 04:00 (runs on ana-nas as lkraven).
 - **nh3-nas → ana-nas** daily at 05:00 (runs on nh3-nas as root —
  rest-server-nh3's container writes files mode 400 owned by admin,
  only root can read them on the Synology side).
 Tracked at `configs/rsync/{ana-nas-to-nh3,nh3-nas-to-ana}/`.
 ## Governing principles (durable rules)
 Saved in feedback-type memory files; apply by default.
 1. **Backups must not risk production.** If a backup process could
   crash the host it's backing up, change the strategy or skip the
   backup. Reference: CT 109 (ana-nas) self-backup crash of
   2026-04-23. Memory: `feedback_backups_must_not_risk_production.md`.
 2. **DB data lives on local disk, not NFS.** Decision 2026-04-23
   after pfi-postgres migration off `/mnt/db`. New DB deployments
   default to local storage; shared NFS for DB data is an anti-pattern
   in this environment. Memory: `project_db_migrate_off_nfs.md`.
 3. **Use `ssh -t` when the remote runs sudo.** Read-only probes and
   piped scripts can stay plain `ssh`. Memory: `feedback_ssh_sudo.md`.
 4. **Handle git commands autonomously.** Stage + commit at natural
   boundaries without asking; verify risky changes first; never
   force-push or push without explicit approval. Memory:
   `feedback_git_autonomous.md`.
 5. **No Claude attribution in commits.** No `Co-Authored-By`,
   "Generated with...", robot emojis, etc. Memory:
   `feedback_git_commits.md`.
 ## Known gotchas (save yourself time)
 These caught us once; don't let them catch you twice.
 ### NFS / storage
 - **ana-nas is `CT 109` (LXC), not `VM 100`.** Persistent confusion
  historically — the Proxmox VM name was `PFI-ANA-TRUENAS` for ages
  but the host was reprovisioned to Debian, and it's an LXC container
  not a VM. Memory: `storage_ana_nas.md`.
 - **ana-nas is a SPOF for rest-server-ana + PBS-ANA datastore.** When
  multiple fleet services fail together, check `ping 10.250.50.50`
  FIRST. Memory: `incident_ana_nas_spof.md`. Recovery runbook:
  `docs/runbooks/disaster-recovery.md`.
 - **NFS root_squash bites in two directions:**
  - **Sudo-root on the client writing to an NFS file** gets mapped to
    nobody. Doesn't matter who owns the file.
  - **rsync-as-root of NFS-owned data** fails the same way. Workaround
    for postgres-on-NFS migration: run rsync as the `postgres` user,
    whose uid has legitimate NFS access.
 - **ZFS with `casesensitivity=insensitive` + NFSv4 = EACCES on writes.**
  PBS-ANA's datastore uses NFSv3 mount to work around this. Don't
  "upgrade" to NFSv4 without fixing the dataset first.
 - **Synology `.htpasswd` edits must be done as the file owner** (uid
  1000 = lkraven for rest-server-ana), NOT via sudo. sudo-root gets
  squashed to nobody and can't read/write the file.
 - **Synology Advanced Permissions (syno_acl)** imposes NFSv4 ACLs that
  shadow POSIX perms. "Enable Advanced Permissions = off" doesn't
  reset it once set. Fix for PBS data: `chmod 777` on the share to
  flatten to "Linux mode" (confirm with `synoacltool -get`).
 - **DSM Container Manager** doesn't expose `sudo docker` in root's
  PATH. Use DSM web UI to restart containers, or find the actual
  binary via `find`.
 - **scp to DSM** requires `-O` flag (legacy scp protocol) — default
  sftp subsystem is disabled.
 ### systemd
 - **DSM ships a pre-v247 systemd.** Refuses `Restart=on-failure` on
  `Type=oneshot`, doesn't recognize `StartLimitBurst/IntervalSec`,
  `systemctl --now` flag absent. Write unit files with plain oneshot
  semantics for DSM; the daily timer retry compensates for missed
  runs.
 - **fstab with bare `defaults`** doesn't auto-retry a failed NFS
  mount. If the NFS server wasn't up at boot, `mnt-*.mount` sits in
  failed state until manual `systemctl reset-failed` + `systemctl start`
  (or simpler: `mount -a`).
 - **Ghost files on NFS mount points.** When NFS unmounts while writes
  are still landing at the mount point path, the writes go to the
  underlying local directory. When NFS remounts, the local ghost is
  shadowed. When NFS later unmounts again, the ghost reappears and
  breaks whatever was reading that path. See `incident_ana_nas_spof.md`
  for the exact case on ana-docker.
 ### Naming / identifiers
 - **Hardware vs OS distinction** for servers with iDRAC/BMC:
  - `pfi-pve` (Dell R750xs) ↔ `PFI-r750xs-iDRAC` at 10.250.250.30
  - `sfsrv-ana` (Dell R630) ↔ `SF-R630-iDRAC` at 10.250.250.110
  - `ana-ml2` (Supermicro) ↔ `PFI-ANA-ML2 BMC` at 10.250.250.50
  Each `servers/<name>/` dir documents the OS side; hardware/iDRAC
  entries exist in homepage but are separate logical identities.
 - **irv-ml1 was `ana-ml1`** before a physical move; OS hostname still
  says `ana-ml1` pending an explicit rename. Doesn't affect services.
 ### Workflow
 - **Terminal word-wrap breaks long pasted commands.** Never embed a
  580-char pubkey or long URL inline in a multi-line shell command —
  terminal wraps the line, shell captures literal newlines, and the
  command fails in bizarre ways. Use `cat` from a file, `printf`, or
  a tempfile + scp.
 - **zsh interactive mode doesn't parse `#` as comment by default.**
  Paste multi-line blocks with comment lines and zsh tries to execute
  `#` as a command. Either tell user to `setopt interactivecomments`
  or keep blocks comment-free.
 - **`ssh -t` with piped stdin fails** — `-t` allocates a tty, stdin
  from a pipe can't be a tty. Options: don't use `-t` (give up
  interactive sudo prompt), or scp the file first then `ssh -t` for
  the sudo bits.
 ## Where to look for what
 | Need | Go here |
 |---|---|
 | Is host X alive? | `servers/<host>/system-details.txt` (snapshot). Refresh with `scripts/refresh-server-info.sh <host>`. |
 | What does host X run? | `servers/<host>/README.md` |
 | What's the backup strategy for host X? | `configs/restic/<host>/README.md` + `profiles.yaml` |
 | How do I recover from host X going down? | `docs/runbooks/disaster-recovery.md` |
 | How was PBS deployed? | `docs/runbooks/pbs-deployment.md` |
 | What's currently open / in-flight? | `STATUS.md` |
 | What do I need to know that isn't in current code? | `MEMORY.md` + the `.md` files it links |
 | Why did we do X? | Check memory files + `STATUS.md` session milestones at the bottom |
 ## Inventory + automation scripts
 Under `scripts/`:
 - `server_inspect.sh` — pipes over SSH to generate `system-details.txt`
  (Docker hosts mostly; works on any Debian-ish Linux).
 - `refresh-server-info.sh <host>` or `all` — fleet-wide refresh
  driver. `--validate-only` for dry-run.
 - `proxmox_inspect.sh` + `refresh-proxmox-info.sh` — Proxmox-flavored
  variant that captures `qm list`, `pct list`, zpool, storage.
 - `discover-fortigate.sh`, `discover-unifi.sh`, `discover-gaps.sh` —
  network-level inventory diff against `servers/*/`.
 - `sync-stacks.sh` / `deploy-stack.sh` — compose mirror pull/push.
 - `add-host.sh <name> <user@ip>` — canonical host registration.
 ## Current state (2026-04-23 snapshot)
 - **Host inventory:** ~22 tracked hosts under `servers/` across ANA,
  NH3, ESH, IRV. Every host has SSH config alias for convenience.
 - **Backup coverage:** 8 restic clients, 5 hypervisors on PBS with
  DR mirror, cross-site rsync both directions operational.
 - **Recent migrations:** pfi-postgres → local disk (removes ana-nas
  cascade); offen sidecars retired on esh-docker-vm (16 GB reclaimed);
  irv-ml1 + esh-vm-db fully onboarded.
 - **Recent secret rotations:** 6 credentials — all gitea/vaultwarden/
  paperless DB passwords + 3 rest-server htpasswd + repo passphrases.
 Active watchlist (see `STATUS.md` for full detail):
 - pfi-pve backplane repair pending tech visit
 - Postgres NFS retirement (block 4+5) pending 24h burn-in
 - Tomorrow 01:00: first scheduled restic runs for esh-vm-db + irv-ml1
 - PBS Phase 7-8 (retire legacy vzdump) gated on 2026-04-29
 ## First commands a fresh session should be comfortable running
 ```bash
 # Fleet validation (read-only, safe)
 scripts/refresh-server-info.sh --validate-only all
 # Specific host refresh
 scripts/refresh-server-info.sh <hostname>
 # Look at current fleet state
 cat STATUS.md
 # Explore memory
 ls /home/lkraven/.claude/projects/-home-lkraven-development-eshpfi-management/memory/
 cat /home/lkraven/.claude/projects/-home-lkraven-development-eshpfi-management/memory/MEMORY.md
 # Probe any host
 ssh <host>  # alias resolves via ~/.ssh/config
 ```
 ## When in doubt
 - Read `CLAUDE.md` for conventions and the Servers table.
 - Read `STATUS.md` for what's in-flight right now.
 - Check `memory/MEMORY.md` for durable facts about the user's
  preferences + project history.
 - For recovery questions: `docs/runbooks/disaster-recovery.md`.
 - Ask. User is experienced — doesn't need hand-holding, but a clear
  question is faster than a wrong assumption.
@@ -0,0 +1,78 @@
 ===== HOST =====
 Hostname:   ana-filebot
 Date:       2026-04-23T08:04:59+00:00
 Uptime:     up 34 weeks, 4 days, 7 hours, 50 minutes
 OS:         Debian GNU/Linux 12 (bookworm)
 Kernel:     6.8.12-8-pve
 Arch:       x86_64
 ===== HARDWARE =====
 CPU cores:  4
 CPU model:  Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz
 MemTotal:   2.0 GB
 MemAvailable: 2.0 GB
 ===== GPUS =====
 nvidia-smi not present (no NVIDIA GPUs or driver not installed)
 ===== FILESYSTEMS (df) =====
 Filesystem                Size  Used Avail Use% Mounted on
 ospool/subvol-112-disk-0   80G  813M   80G   1% /
 ospool/backup              11T  240M   11T   1% /mnt/ospool-backup
 NASPool/backupStore        20T   70G   20T   1% /mnt/backup
 NASPool/db                 20T  282M   20T   1% /mnt/db
 NASPool/docker             20T  384K   20T   1% /mnt/docker
 NASPool/pve-VMStorage      22T  2.4T   20T  12% /mnt/pve-VMStorage
 NASPool/webdav             20T  256K   20T   1% /mnt/webdav
 ===== PERSISTENT MOUNTS (/etc/fstab, non-comment) =====
 ===== TARGETED DATA PATHS =====
 /opt  (total: 512)
  total 9
  drwxr-xr-x  2 root root  2 2023-10-10 13:26 .
  drwxr-xr-x 17 root root 21 2025-08-24 00:14 ..
 /srv  (total: 512)
  total 9
  drwxr-xr-x  2 root root  2 2023-10-10 13:26 .
  drwxr-xr-x 17 root root 21 2025-08-24 00:14 ..
 ===== DOCKER =====
 docker not installed
 ===== COMPOSE FILES (/opt/docker/compose/) =====
 /opt/docker/compose not present
 ===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) =====
 /opt/docker/conf not present
 ===== LISTENING PORTS =====
 *:22
 *:8000
 127.0.0.1:25
 [::1]:25
 ===== MODEL / HUGGINGFACE CACHES =====
 ===== DOCKER-ADJACENT SYSTEMD SERVICES =====
 container-getty@1.service running
 container-getty@2.service running
 ===== DONE =====
 Paste the above back into the chat, or pass a path as argv[1] to save.
@@ -4,11 +4,15 @@ Primary AI inference host for PFI.
 ## Network
- **LAN IP:** 10.250.50.54
+- **LAN IP:** 10.250.50.54 (in-band, OS-side)
- **SSH:** standard port 22
+- **BMC (OOB):** 10.250.250.50 — Supermicro IPMI web UI
  at <https://10.250.250.50> (homepage card: *PFI-ANA-ML2 BMC*)
 - **SSH:** standard port 22 on 10.250.50.54
 ## Hardware
 - **Chassis:** Supermicro mid-range inferencing server (bare metal,
  NOT Dell / not the same box as sf-r630 / sfsrv-ana)
 - **CPU:** AMD EPYC 9254 24-core (96 threads)
 - **RAM:** 566 GB
 - **GPUs:** 2x NVIDIA RTX 6000 Ada Generation (46 GB VRAM each, GPU 0 and GPU 1)
@@ -0,0 +1,77 @@
 # ana-nas
 Debian NFS/SMB file server at the Anaheim site — **CT 109** (LXC) on
 pfi-pve. Serves the fleet's primary storage layer:
 - `/mnt/backup` → restic target for `rest-server-ana` on ana-docker
  + backing storage for PBS-ANA's datastore
 - `/mnt/docker` → compose/config shares
 - `/mnt/webdav` → WebDAV data (if still active)
 - Historically `/mnt/db` (retired 2026-04-23 after the DB-off-NFS
  migration; pfi-postgres now uses local disk)
 ## Network
 - **LAN IP:** 10.250.50.50
 - **SSH:** `ssh ana-nas` (config alias → `lkraven@10.250.50.50`,
  key auth). lkraven is a standard user — sudo for root operations.
 ## Container context
 This is **CT 109**, not a VM. Bind-mounts into the LXC from pfi-pve's
 `ospool` ZFS datasets:
 | Inside CT | Host path on pfi-pve | Purpose |
 |---|---|---|
 | `/mnt/ospool-backup` | ZFS dataset | ospool snapshots staging |
 | `/mnt/backup` | ZFS dataset | rest-server-ana + PBS-ANA data |
 | `/mnt/db` | ZFS dataset | **retired 2026-04-23** (postgres moved to local disk) |
 | `/mnt/docker` | ZFS dataset | compose files + conf shares |
 | `/mnt/pve-VMStorage` | ZFS dataset | alt VM storage pool |
 | `/mnt/webdav` | ZFS dataset | WebDAV data |
 Because these are bind-mounts, CT 109's own vzdump backup (LXC rootfs
 only) does NOT capture the data payload — the data lives on pfi-pve's
 ospool and is captured via the host's own backup chain.
 ## Why CT 109 matters (SPOF warning)
 As of 2026-04-23, ana-nas still affects these fleet services if it
 goes offline:
 - **rest-server-ana** (on ana-docker) — its data dir is NFS-mounted
  from here; an outage kills file-level backups fleet-wide
 - **PBS-ANA datastore** — NFS-mounted from here; an outage kills
  VM-level backups fleet-wide
 - **Cross-site restic rsync source** (ana-nas → nh3-nas daily)
 **Not** affected anymore (as of 2026-04-23 DB migration):
 - pfi-postgres (VM 105) — now on local disk, ana-nas outages
  don't cascade into vaultwarden/gitea/paperless/zammad
 Full recovery procedure: `memory/incident_ana_nas_spof.md` +
 `docs/runbooks/disaster-recovery.md`.
 ## DSM / rescue access
 Under the hood this is vanilla Debian 12 with hand-configured NFS/SMB.
 No appliance tooling — share tweaks go through `/etc/exports` +
 `systemctl reload nfs-server`. Historical confusion about this box
 being a TrueNAS appliance persists in naming (the Proxmox label was
 `PFI-ANA-TRUENAS` for a long time) but it is plain Debian.
 ## History note
 ID-over-time: this host has been called VM 100, PFI-ANA-TRUENAS,
 and now CT 109 / ana-nas. Memory file `storage_ana_nas.md` records
 the VM→CT correction that was clarified 2026-04-23 from the vzdump
 log output.
 ## Refresh
 ```
 scripts/refresh-server-info.sh ana-nas
 ```
 Snapshot at `system-details.txt`.
@@ -0,0 +1 @@
 lkraven@10.250.50.50
@@ -0,0 +1,96 @@
 ===== HOST =====
 Hostname:   ana-nas
 Date:       2026-04-23T06:53:27+00:00
 Uptime:     up 34 weeks, 3 days, 11 hours, 22 minutes
 OS:         Debian GNU/Linux 12 (bookworm)
 Kernel:     6.8.12-8-pve
 Arch:       x86_64
 ===== HARDWARE =====
 CPU cores:  4
 CPU model:  Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz
 MemTotal:   2.0 GB
 MemAvailable: 1.9 GB
 ===== GPUS =====
 nvidia-smi not present (no NVIDIA GPUs or driver not installed)
 ===== FILESYSTEMS (df) =====
 Filesystem                Size  Used Avail Use% Mounted on
 ospool/subvol-109-disk-0   80G  813M   80G   1% /
 ospool/backup              11T  240M   11T   1% /mnt/ospool-backup
 NASPool/backupStore        20T   67G   20T   1% /mnt/backup
 NASPool/db                 20T  282M   20T   1% /mnt/db
 NASPool/docker             20T  384K   20T   1% /mnt/docker
 NASPool/pve-VMStorage      22T  2.4T   20T  12% /mnt/pve-VMStorage
 NASPool/webdav             20T  256K   20T   1% /mnt/webdav
 ===== PERSISTENT MOUNTS (/etc/fstab, non-comment) =====
 ===== TARGETED DATA PATHS =====
 /opt  (total: 512)
  total 9
  drwxr-xr-x  2 root root  2 2023-10-10 13:26 .
  drwxr-xr-x 17 root root 21 2025-08-24 19:31 ..
 /srv  (total: 512)
  total 9
  drwxr-xr-x  2 root root  2 2023-10-10 13:26 .
  drwxr-xr-x 17 root root 21 2025-08-24 19:31 ..
 ===== DOCKER =====
 docker not installed
 ===== COMPOSE FILES (/opt/docker/compose/) =====
 /opt/docker/compose not present
 ===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) =====
 /opt/docker/conf not present
 ===== LISTENING PORTS =====
 *:22
 *:9090
 0.0.0.0:111
 0.0.0.0:139
 0.0.0.0:2049
 0.0.0.0:39347
 0.0.0.0:40609
 0.0.0.0:445
 0.0.0.0:48585
 0.0.0.0:49363
 0.0.0.0:57311
 127.0.0.1:25
 [::1]:25
 [::]:111
 [::]:139
 [::]:2049
 [::]:41319
 [::]:445
 [::]:47023
 [::]:50729
 [::]:50951
 [::]:55887
 ===== MODEL / HUGGINGFACE CACHES =====
 ===== DOCKER-ADJACENT SYSTEMD SERVICES =====
 container-getty@1.service running
 container-getty@2.service running
 ===== DONE =====
 Paste the above back into the chat, or pass a path as argv[1] to save.
@@ -0,0 +1,73 @@
 ===== HOST =====
 Hostname:   ana-wg
 Date:       2026-04-23T08:05:00+00:00
 Uptime:     up 34 weeks, 4 days, 7 hours, 50 minutes
 OS:         Debian GNU/Linux 12 (bookworm)
 Kernel:     6.8.12-8-pve
 Arch:       x86_64
 ===== HARDWARE =====
 CPU cores:  4
 CPU model:  Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz
 MemTotal:   2.0 GB
 MemAvailable: 2.0 GB
 ===== GPUS =====
 nvidia-smi not present (no NVIDIA GPUs or driver not installed)
 ===== FILESYSTEMS (df) =====
 Filesystem                Size  Used Avail Use% Mounted on
 ospool/subvol-113-disk-0  8.0G  623M  7.4G   8% /
 ===== PERSISTENT MOUNTS (/etc/fstab, non-comment) =====
 ===== TARGETED DATA PATHS =====
 /opt  (total: 1.0K)
  total 10
  drwxr-xr-x  2 root root  3 2024-10-30 18:08 .
  drwxr-xr-x 17 root root 21 2025-08-24 00:14 ..
  lrwxrwxrwx  1 root root 29 2024-10-30 18:08 pivpn -> /usr/local/src/pivpn/scripts/
 /srv  (total: 512)
  total 9
  drwxr-xr-x  2 root root  2 2023-10-10 13:26 .
  drwxr-xr-x 17 root root 21 2025-08-24 00:14 ..
 ===== DOCKER =====
 docker not installed
 ===== COMPOSE FILES (/opt/docker/compose/) =====
 /opt/docker/compose not present
 ===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) =====
 /opt/docker/conf not present
 ===== LISTENING PORTS =====
 0.0.0.0:22
 127.0.0.1:25
 [::1]:25
 [::]:22
 ===== MODEL / HUGGINGFACE CACHES =====
 ===== DOCKER-ADJACENT SYSTEMD SERVICES =====
 container-getty@1.service running
 container-getty@2.service running
 ===== DONE =====
 Paste the above back into the chat, or pass a path as argv[1] to save.
@@ -0,0 +1,96 @@
 # esh-vm-db
 ESH-side database VM at `esteban.net`. Hosts **two database servers**:
 - **PostgreSQL 15** on port 5432 — primary consumer is paperless-ngx
  on esh-docker-vm (`paperless-ng` user/db).
 - **MongoDB** on port 27017 — consumers unknown; document at next
  inventory pass.
 Discovered 2026-04-23 during the paperless-ngx password rotation —
 paperless's `PAPERLESS_DBHOST: 10.0.50.60` pointed here rather than
 the assumed pfi-postgres (VM 105). Up until that point this host was
 invisible to inventory.
 ## Network
 - **LAN IP:** 10.0.50.60
 - **FQDN:** `esh-vm-db.esteban.net`
 - **SSH:** `ssh esh-vm-db` (config alias → `lkraven@10.0.50.60`,
  key auth).
 - **Listening ports:** 22 (ssh), 111 (rpcbind — NFS mount support),
  5432 (postgres), 27017 (mongo).
 ## Hardware
 - **Hypervisor:** presumably esh-pve or esh-pve-nas (to be confirmed
  — not yet queried from the PVE side).
 - **CPU:** Intel Core i9-13900H (16 cores, mobile) — indicates this
  VM lives on a mini-PC / NUC-class ESH host.
 - **RAM:** 7.8 GB
 - **Root disk:** 250 GB ext4 on `/dev/sda1`, 5.1 GB used (2%). Plenty
  of headroom for both DBs.
 - **OS:** Debian 12 (bookworm), kernel 6.1.0-23
 - **Uptime:** 3+ weeks at last snapshot.
 ## What it runs
 - **PostgreSQL 15 (apt package, `postgresql`)** — data directory
  default (`/var/lib/postgresql/15/main` presumably — confirm with
  `pg_lsclusters` on a future refresh). Local disk, not NFS —
  consistent with the "DBs must live on local disk" principle we
  adopted 2026-04-23.
 - **MongoDB** — port 27017 bound to 0.0.0.0. DataPath + version to
  be captured.
 - **No Docker.** `server_inspect.sh` reports `docker not installed`
  and `/opt/docker` absent — this host is bare-metal DB service,
  not a container host.
 ## Storage
 - `/dev/sda1` (250 GB ext4) — DB data + OS
 - `/mnt/backup` mounted from `10.0.50.50:/mnt/backup` via NFS
  (`defaults`). Purpose TBD — possibly for pg_dump outputs or other
  backup staging. Review on next pass.
 ## Consumers
 - **paperless-ngx** on esh-docker-vm — uses postgres user
  `paperless-ng` against DB `paperless-ng` (confirmed 2026-04-23,
  password rotated).
 - **MongoDB consumers** — unknown; map these by checking connection
  logs or by inventorying the other ESH-side Docker stacks for
  `27017`/`mongodb` references.
 ## Backup coverage
 **Not yet configured.** Same category as ana-ml2/irv-ml1/esh-vm-db
 now (bare-metal DB host without vzdump awareness from the PVE side).
 Next steps:
 - Add to fleet PBS vzdump schedule (its hypervisor backs it up via
  PBS-ANA if the hypervisor is onboarded).
 - Draft `configs/restic/esh-vm-db/profiles.yaml` for file-level +
  per-DB dumps (`pg_dumpall` for postgres, `mongodump` for mongo).
  Target: `rest-server-ana` via the ESH → ANA cross-WAN path, same
  as other ESH-side restic clients.
 - `rest-server-ana`'s `.htpasswd` will need an `esh-vm-db` entry.
 ## Refresh state
 ```bash
 scripts/refresh-server-info.sh esh-vm-db
 ```
 ## Follow-ups (do soon)
 - Confirm hypervisor (likely esh-pve or esh-pve-nas) — `qm list`
  on each and find VM with this name.
 - Map MongoDB consumers.
 - Configure backup (restic profile + per-DB dumps + vzdump coverage
  on the hypervisor side).
 - Audit postgres `pg_hba.conf` — is 5432 bound to 0.0.0.0 with
  proper host-based auth, or is it over-exposed? paperless connects
  from 10.0.50.45 (esh-docker-vm); other hosts shouldn't be able
  to reach in.
@@ -0,0 +1 @@
 lkraven@10.0.50.60
@@ -0,0 +1,80 @@
 ===== HOST =====
 Hostname:   esh-vm-db.esteban.net
 Date:       2026-04-23T21:53:16-07:00
 Uptime:     up 3 weeks, 2 days, 23 hours, 22 minutes
 OS:         Debian GNU/Linux 12 (bookworm)
 Kernel:     6.1.0-23-amd64
 Arch:       x86_64
 ===== HARDWARE =====
 CPU cores:  16
 CPU model:  13th Gen Intel(R) Core(TM) i9-13900H
 MemTotal:   7.8 GB
 MemAvailable: 7.2 GB
 ===== GPUS =====
 nvidia-smi not present (no NVIDIA GPUs or driver not installed)
 ===== FILESYSTEMS (df) =====
 Filesystem              Size  Used Avail Use% Mounted on
 /dev/sda1               250G  5.1G  233G   3% /
 10.0.50.50:/mnt/backup   92T   11G   92T   1% /mnt/backup
 ===== PERSISTENT MOUNTS (/etc/fstab, non-comment) =====
 UUID=9022f0b6-4299-452a-a4a4-85e1623243ae /               ext4    errors=remount-ro 0       1
 UUID=6d90bc64-6f3e-41b9-8295-640ac854e5e5 none            swap    sw              0       0
 /dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
 10.0.50.50:/mnt/backup /mnt/backup nfs defaults 0 0
 ===== TARGETED DATA PATHS =====
 /opt  (total: 4.0K)
  total 8
  drwxr-xr-x  2 root root 4096 2024-05-20 23:47 .
  drwxr-xr-x 18 root root 4096 2024-08-13 17:35 ..
 /srv  (total: 2.8M)
  total 12
  drwxr-xr-x  3 root root 4096 2024-06-13 10:05 .
  drwxr-xr-x 18 root root 4096 2024-08-13 17:35 ..
  drwxrwxrwx  4 root root 4096 2024-06-13 10:44 backups
 ===== DOCKER =====
 docker not installed
 ===== COMPOSE FILES (/opt/docker/compose/) =====
 /opt/docker/compose not present
 ===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) =====
 /opt/docker/conf not present
 ===== LISTENING PORTS =====
 0.0.0.0:111
 0.0.0.0:22
 0.0.0.0:27017
 0.0.0.0:5432
 [::]:111
 [::]:22
 [::]:5432
 ===== MODEL / HUGGINGFACE CACHES =====
 ===== DOCKER-ADJACENT SYSTEMD SERVICES =====
 (none matching)
 ===== DONE =====
 Paste the above back into the chat, or pass a path as argv[1] to save.
@@ -0,0 +1,144 @@
 # irv-ml1 onboarding checklist
 Steps to bring irv-ml1 fully into the fleet, matching the conventions
 used by the other hosts.
 ## Already done (repo-side)
 - [x] `servers/irv-ml1/` registered, ssh-target + system-details captured
 - [x] SSH config alias `ssh irv-ml1`
 - [x] CLAUDE.md Servers table + Placement rules updated
 - [x] `configs/restic/irv-ml1/profiles.yaml` + README drafted
 - [x] Homepage card under new `Infra - IRV` group (IRV-ML1 ping monitor)
 - [x] Homepage `docker.yaml` entry staged (commented) for Docker auto-discovery
 - [x] Homepage `settings.yaml` — `Infra - IRV` group added to Infrastructure tab
 ## To do on the host
 ### 1. Rename hostname (optional, non-blocking)
 ```bash
 ssh -t irv-ml1 'sudo hostnamectl set-hostname irv-ml1 && \
                sudo sed -i "s/ana-ml1/irv-ml1/g" /etc/hosts && \
                hostname && cat /etc/hostname'
 scripts/refresh-server-info.sh irv-ml1  # capture new identity
 ```
 ### 2. Add lkraven to docker group (blocks server_inspect full output)
 ```bash
 ssh -t irv-ml1 'sudo usermod -aG docker lkraven'
 # then logout + login for group to take effect
 ```
 ### 3. Install Docker + Dockge (your call on convention adoption)
 Once installed, create the fleet-standard directory layout:
 ```bash
 ssh -t irv-ml1 'sudo mkdir -p /opt/docker/{compose,conf} && \
                sudo chown -R lkraven:lkraven /opt/docker'
 ```
 Create the external traefik-net (even if no traefik here yet — stacks
 reference it as `external: true`):
 ```bash
 ssh -t irv-ml1 'docker network create traefik-net'
 ```
 ### 4. Expose Docker socket on port 2375 (for homepage auto-discovery)
 On irv-ml1:
 ```bash
 sudo mkdir -p /etc/systemd/system/docker.service.d
 sudo tee /etc/systemd/system/docker.service.d/override.conf > /dev/null <<'EOF'
 [Service]
 ExecStart=
 ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2375
 EOF
 sudo systemctl daemon-reload && sudo systemctl restart docker
 ```
 Then uncomment the `irv-ml1-docker:` block in
 `configs/homepage/docker.yaml` and push homepage config.
 **Trust model:** 2375 is unencrypted; only safe because it's bound to
 the WG tunnel interface on the remote side. Confirm your firewall
 rules prevent exposure of 2375 on any non-WG interface.
 ### 5. Deploy Dockge
 Standard fleet pattern — see `stacks/dockge/` or copy from
 esh-docker-vm/nh3-docker. Port 5001. After Dockge is up:
 - Uncomment the `IRV-ML1-Dockge` entry in `configs/homepage/services.yaml`
 - Push homepage config
 ### 6. Deploy monitoring agents
 Cross-site Beszel + Dozzle agents report to hubs on ana-docker:
 - `beszel-agent` → ana-docker's Beszel hub
 - `dozzle-agent` → ana-docker's Dozzle hub
 Copy compose files from another host (`stacks/beszel-agent/`,
 `stacks/dozzle-agent/`) and adjust `HUB_URL` / TOKEN / display name
 for irv-ml1.
 ### 7. Set up restic
 Follow `configs/restic/irv-ml1/README.md` exactly:
 1. Create `irv-ml1` htpasswd entry on rest-server-nh3 (not rest-server-ana
   — we target the NH3 side because it's the WG termination endpoint)
 2. Install `/etc/restic/restic.env` + `/etc/restic/password` on irv-ml1
 3. `restic init`
 4. Install resticprofile
 5. Deploy `profiles.yaml`
 6. `resticprofile schedule --all`
 7. First manual backup
 8. Verify snapshot appears in Backrest UI on ana-docker
 ### 8. Push homepage config to make cards live
 After each homepage YAML edit in this repo:
 ```bash
 rsync -av --exclude='.env' --exclude='.env.*' \
  configs/homepage/ esh-docker-vm:/opt/docker/conf/homepage/
 ```
 ## Verification after everything
 ```bash
 # Host reachable through WG
 ssh irv-ml1 hostname
 # Docker up, 2375 listening
 ssh irv-ml1 'docker ps && ss -tlnp | grep 2375'
 # Dockge up
 curl -fsS http://10.100.79.3:5001/ >/dev/null && echo DOCKGE_OK
 # Agents reporting (check from ana-docker)
 ssh ana-docker 'docker exec beszel docker logs 2>&1 | grep irv-ml1 | tail'
 # Restic timer armed
 ssh irv-ml1 'systemctl list-timers "resticprofile*"'
 # First restic snapshot visible
 # -> Backrest UI → repos → irv-ml1
 ```
 ## WireGuard health considerations
 Everything this host does depends on the WG tunnel. Worth adding:
 - Beszel alert on ping loss to 10.100.50.50 (WG peer)
 - Or a simple cron on ana-docker that pings 10.100.79.3 every 5 min
  and alerts via mailrise on failure
 Not blocking onboarding, but the tunnel is the hidden SPOF for every
 irv-ml1 service.
@@ -0,0 +1 @@
 lkraven@10.100.79.3
@@ -0,0 +1,75 @@
 # nh3-nas
 Synology RS2418+ at the NH3 site (`PFI-NH3-NAS`, `10.100.50.50`).
 Primary role is fleet storage: VM-image hosting for nh3-pve, the
 NH3-side restic target, and as of 2026-04-22 the PBS-NH3 datastore
 backend.
 ## Network
 - **LAN IP:** 10.100.50.50
 - **DSM web UI:** <https://10.100.50.50:5001>
 - **SSH:** `ssh nh3-nas` (config alias → `syncuser@10.100.50.50`,
  key auth). The `syncuser` account was created specifically for
  automation/tooling access; `admin` retained as the DSM primary.
 ## Hardware
 - **Model:** Synology RS2418+ (denverton platform)
 - **CPU:** Intel Atom C3538 @ 2.10 GHz, 4 cores
 - **RAM:** 31.3 GB
 - **Kernel:** `4.4.302+` — DSM's custom kernel, not a vanilla Debian
  base. Implications: very old bash features only (4.x), BusyBox-ish
  userland for some tools, syno_acl layer on Btrfs.
 - **Volume:** `/volume1`, Btrfs on `cachedev_0` (SSD-cached LVM).
  42 TB total, 27 TB used (64%), 16 TB free.
 ## Services running on the box
 - **DSM built-in NFS server** — exports under `/etc/exports` managed
  via DSM Shared Folder UI. Active exports include:
  - `/volume1/Shared`, `/volume1/Media`, `/volume1/NetBackup`,
    `/volume1/Backup`, `/volume1/compose`, `/volume1/VMStorage`,
    `/volume1/devstor` (legacy home-lab exports; `all_squash`)
  - `/volume1/pbs` — dedicated share for PBS-NH3's datastore mount.
    Linux/POSIX mode 777, no syno_acl, `no_root_squash + no_all_squash`.
    Do NOT enable Advanced Permissions on this share — see
    `docs/runbooks/pbs-deployment.md` Phase 5.3 for the history.
 - **rest-server-nh3** (Docker via DSM ContainerManager) — restic
  HTTP endpoint on port 8000, writes under `/volume1/Backup`. Serves
  nh3-docker and nh3-dev restic clients.
 ## What backs up to it
 - nh3-docker + nh3-dev resticprofile timers (via rest-server-nh3)
 - PBS-NH3 datastore (`backups-mirror`), which receives the nightly
  sync pull from PBS-ANA
 ## What backs up FROM it
 Not currently backed up itself — the DSM side is the source-of-truth
 for its own config. Future work: cross-site rsync of
 `/volume1/Backup/restic/...` to the Ana NAS (`10.250.50.50`), blocked
 previously on SSH-access to this host (now resolved with `syncuser`).
 ## DSM-specific gotchas
 - **Docker:** runs via DSM's ContainerManager package, not a native
  `docker` CLI. `docker ps` over SSH returns empty; use
  `sudo synopkg list | grep -i container` and
  `sudo docker ...` (DSM aliases the binary into root's PATH).
 - **Home directories:** live under `/var/services/homes/<user>/` (not
  `/home/<user>`). SSH public keys go in `~/.ssh/authorized_keys`
  which maps to the DSM home. Home dir permission must be 755 (not
  700) or sshd rejects keys silently after DSM updates reset it.
 - **Btrfs mount options** include `synoacl` — POSIX permissions are
  projected through Synology's ACL layer. For share permissions to
  behave as plain POSIX, either turn off "Advanced Permissions" in
  DSM *and* flatten with `chmod 777` (which converts the share to
  "Linux mode", `synoacltool -get` will confirm), or grant explicit
  ACL entries with `synoacltool -add`.
 ## Refresh state
 Snapshot captured via `scripts/refresh-server-info.sh nh3-nas` —
 updates `system-details.txt` alongside this README.
@@ -0,0 +1 @@
 syncuser@10.100.50.50
@@ -0,0 +1,126 @@
 ===== HOST =====
 Hostname:   PFI-NH3-NAS
 Date:       2026-04-22T23:46:21-07:00
 Uptime:     up 3 days, 9 hours, 26 minutes
 Kernel:     4.4.302+
 Arch:       x86_64
 ===== HARDWARE =====
 CPU cores:  4
 CPU model:  Intel(R) Atom(TM) CPU C3538 @ 2.10GHz
 MemTotal:   31.3 GB
 MemAvailable: 29.5 GB
 ===== GPUS =====
 nvidia-smi not present (no NVIDIA GPUs or driver not installed)
 ===== FILESYSTEMS (df) =====
 Filesystem              Size  Used Avail Use% Mounted on
 /dev/md0                2.3G  1.5G  749M  67% /
 /dev/mapper/cachedev_0   42T   27T   16T  64% /volume1
 /dev/loop0               27M  767K   24M   4% /tmp/SynologyAuthService
 /dev/mapper/cachedev_0   42T   27T   16T  64% /volume1/@appdata/ContainerManager/all_shares/Backup
 /dev/mapper/cachedev_0   42T   27T   16T  64% /volume1/@appdata/ContainerManager/all_shares/compose
 /dev/mapper/cachedev_0   42T   27T   16T  64% /volume1/@appdata/ContainerManager/all_shares/devstor
 /dev/mapper/cachedev_0   42T   27T   16T  64% /volume1/@appdata/ContainerManager/all_shares/docker
 /dev/mapper/cachedev_0   42T   27T   16T  64% /volume1/@appdata/ContainerManager/all_shares/homes
 /dev/mapper/cachedev_0   42T   27T   16T  64% /volume1/@appdata/ContainerManager/all_shares/Media
 /dev/mapper/cachedev_0   42T   27T   16T  64% /volume1/@appdata/ContainerManager/all_shares/Music
 /dev/mapper/cachedev_0   42T   27T   16T  64% /volume1/@appdata/ContainerManager/all_shares/NetBackup
 /dev/mapper/cachedev_0   42T   27T   16T  64% /volume1/@appdata/ContainerManager/all_shares/Shared
 /dev/mapper/cachedev_0   42T   27T   16T  64% /volume1/@appdata/ContainerManager/all_shares/VMStorage
 /dev/mapper/cachedev_0   42T   27T   16T  64% /volume1/@appdata/ContainerManager/all_shares/web
 /dev/mapper/cachedev_0   42T   27T   16T  64% /volume1/@appdata/ContainerManager/all_shares/web_packages
 /dev/mapper/cachedev_0   42T   27T   16T  64% /volume1/@appdata/ContainerManager/all_shares/pbs
 ===== PERSISTENT MOUNTS (/etc/fstab, non-comment) =====
 none /proc proc defaults 0 0
 /dev/root / ext4 defaults 1 1
 /dev/mapper/cachedev_0 /volume1 btrfs auto_reclaim_space,ssd,synoacl,relatime,nodev 0 0
 ===== TARGETED DATA PATHS =====
 /opt  (total: 4.0K)
 ===== DOCKER =====
 docker not installed
 ===== COMPOSE FILES (/opt/docker/compose/) =====
 /opt/docker/compose not present
 ===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) =====
 /opt/docker/conf not present
 ===== LISTENING PORTS =====
 0.0.0.0:111
 0.0.0.0:139
 0.0.0.0:161
 0.0.0.0:2022
 0.0.0.0:2049
 0.0.0.0:22
 0.0.0.0:4045
 0.0.0.0:443
 0.0.0.0:445
 0.0.0.0:5000
 0.0.0.0:5001
 0.0.0.0:5357
 0.0.0.0:662
 0.0.0.0:80
 0.0.0.0:8000
 0.0.0.0:873
 0.0.0.0:892
 10.100.250.50:3260
 10.100.50.50:3260
 127.0.0.1:33304
 127.0.0.1:512
 127.0.0.1:5432
 :::111
 :::139
 :::161
 :::2022
 :::2049
 :::22
 :::22000
 :::3261
 :::3263
 :::3264
 :::3265
 :::4045
 :::443
 :::445
 :::5000
 :::5001
 :::5357
 :::6281
 :::662
 :::80
 :::8000
 :::8384
 :::873
 :::892
 fe80::211:32ff:fe9:3260
 fe80::ee0d:9aff:fe:3260
 ===== MODEL / HUGGINGFACE CACHES =====
 ===== DOCKER-ADJACENT SYSTEMD SERVICES =====
 pkg-ContainerManager-dockerd.service running
 pkg-ContainerManager-event-watcherd.service running
 pkg-ContainerManager-termd.service running
 ===== DONE =====
 Paste the above back into the chat, or pass a path as argv[1] to save.
@@ -0,0 +1,87 @@
 # pbs-ana
 Proxmox Backup Server — **fleet primary**. VM on pfi-pve with an
 NFS-mounted datastore on the Ana NAS. All 5 hypervisors (pfi-pve,
 nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) back up here; one-way sync
 to pbs-nh3 at 06:00 daily.
 ## Network
 - **LAN IP:** 10.250.50.90
 - **FQDN:** `pbs-ana.phasefinal.com`
 - **PBS web UI:** <https://10.250.50.90:8007/>
 - **SSH:** `ssh pbs-ana` (config alias → `lkraven@10.250.50.90`).
  Root ops via sudo from lkraven.
 - **Homepage card:** *PBS-ANA* in `Infra - ANA` group.
 ## Hardware (VM)
 - **Hypervisor:** pfi-pve
 - **vCPU:** 4 (QEMU)
 - **RAM:** 7.8 GB
 - **Root disk:** `/dev/sda1` 31 GB ext4 — OS only (2 GB used)
 - **Datastore:** **NFS from ana-nas** at `/mnt/pbs-datastore` →
  `10.250.50.50:/mnt/backup/pbs-ana`, 20 TB (currently 336 GB used)
 ## NFS details (the ZFS-case-insensitivity gotcha)
 The ZFS dataset backing `/mnt/backup` on ana-nas is
 `casesensitivity=insensitive`, which breaks NFSv4 writes (EACCES).
 Mount must be **NFSv3** — that's what `/etc/fstab` already uses:
 ```
 10.250.50.50:/mnt/backup/pbs-ana /mnt/pbs-datastore nfs \
  defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=3 0 0
 ```
 Don't upgrade this to vers=4 without fixing the dataset. See
 `docs/runbooks/pbs-deployment.md` Phase 0.5 for the full history.
 ## What runs here
 - `proxmox-backup-proxy` — HTTPS API + web UI (port 8007)
 - `proxmox-backup` — backend service
 - Scheduled verify job: weekly Saturday 23:00 (`ignore-verified=true`,
  `outdated-after=30 days`)
 - Garbage collection: default weekly
 ## Namespaces
 Per-hypervisor namespaces to prevent VMID collisions across PVE
 hosts (VMIDs are NOT globally unique):
 - `pfi-pve`
 - `nh3-pve`
 - `esh-pve`
 - `esh-pve-nas`
 - `sfsrv-ana`
 ## API token (for hypervisor clients)
 - `root@pam!fleet-vzdump` — stored in password manager; referenced
  by each hypervisor's `/etc/pve/storage.cfg` `pbs-ana` stanza.
 ## Sync to DR
 One-way pull: **pbs-nh3** pulls from here at 06:00 daily. pbs-ana
 does not push; all coordination is on the DR side.
 ## Dependencies / blast radius
 - **Hard dependency on ana-nas** (CT 109). Datastore NFS goes dead
  if ana-nas crashes. Fleet-wide vzdump + verify + sync all fail
  until NFS recovers. See `memory/incident_ana_nas_spof.md`.
 - **Self-backup excluded** from scheduled vzdump — backing up a
  PBS VM to itself is a circular dependency. Config-only backup
  could be captured via file-level restic if needed; not currently
  set up (low priority — the datastore is on NFS, OS is
  reinstallable in ~30 min).
 ## Refresh
 ```
 scripts/refresh-server-info.sh pbs-ana
 ```
 Snapshot at `system-details.txt`. Full deployment history:
 `docs/runbooks/pbs-deployment.md`.
@@ -0,0 +1 @@
 lkraven@10.250.50.90
@@ -0,0 +1,84 @@
 ===== HOST =====
 Hostname:   pbs-ana.phasefinal.com
 Date:       2026-04-23T21:55:34-07:00
 Uptime:     up 7 hours, 26 minutes
 OS:         Debian GNU/Linux 12 (bookworm)
 Kernel:     6.1.0-44-amd64
 Arch:       x86_64
 ===== HARDWARE =====
 CPU cores:  4
 CPU model:  QEMU Virtual CPU version 2.5+
 MemTotal:   7.8 GB
 MemAvailable: 7.4 GB
 ===== GPUS =====
 nvidia-smi not present (no NVIDIA GPUs or driver not installed)
 ===== FILESYSTEMS (df) =====
 Filesystem                        Size  Used Avail Use% Mounted on
 /dev/sda1                          31G  2.2G   27G   8% /
 10.250.50.50:/mnt/backup/pbs-ana   20T  336G   19T   2% /mnt/pbs-datastore
 ===== PERSISTENT MOUNTS (/etc/fstab, non-comment) =====
 UUID=6a2b24f3-4134-40ed-9b64-f708dcb77cef /               ext4    errors=remount-ro 0       1
 UUID=bfe22def-87a6-4ca8-8f06-19bbf148559c none            swap    sw              0       0
 /dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
 10.250.50.50:/mnt/backup/pbs-ana  /mnt/pbs-datastore  nfs  defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=3  0  0
 ===== TARGETED DATA PATHS =====
 /opt  (total: 4.0K)
  total 8
  drwxr-xr-x  2 root root 4096 2026-04-21 21:22 .
  drwxr-xr-x 18 root root 4096 2026-04-21 21:25 ..
 /srv  (total: 4.0K)
  total 8
  drwxr-xr-x  2 root root 4096 2026-04-21 21:22 .
  drwxr-xr-x 18 root root 4096 2026-04-21 21:25 ..
 ===== DOCKER =====
 docker not installed
 ===== COMPOSE FILES (/opt/docker/compose/) =====
 /opt/docker/compose not present
 ===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) =====
 /opt/docker/conf not present
 ===== LISTENING PORTS =====
 0.0.0.0:111
 0.0.0.0:22
 0.0.0.0:36203
 0.0.0.0:36419
 [::]:111
 [::1]:25
 127.0.0.1:25
 127.0.0.1:82
 [::]:22
 [::]:35545
 [::]:56445
 *:8007
 ===== MODEL / HUGGINGFACE CACHES =====
 ===== DOCKER-ADJACENT SYSTEMD SERVICES =====
 (none matching)
 ===== DONE =====
 Paste the above back into the chat, or pass a path as argv[1] to save.
@@ -0,0 +1,99 @@
 # pbs-nh3
 Proxmox Backup Server — **DR mirror**. VM on nh3-pve with an
 NFS-mounted datastore on the NH3 Synology. Pulls nightly from pbs-ana
 (the primary) at 06:00.
 ## Network
 - **LAN IP:** 10.100.50.90
 - **FQDN:** `pbs-nh3.phasefinal.com`
 - **PBS web UI:** <https://10.100.50.90:8007/>
 - **SSH:** `ssh pbs-nh3` (config alias → `lkraven@10.100.50.90`).
  Root ops via sudo from lkraven.
 - **Homepage card:** *PBS-NH3* in `Infra - NH3` group.
 ## Hardware (VM)
 - **Hypervisor:** nh3-pve
 - **vCPU:** 4 (QEMU)
 - **RAM:** 7.8 GB
 - **Root disk:** `/dev/sda1` 31 GB ext4 — OS only (2 GB used)
 - **Datastore:** **NFS from nh3-nas** at `/mnt/pbs-datastore` →
  `10.100.50.50:/volume1/pbs`. Volume size 42 TB; 27 TB used (64%).
 ## NFS details (the Synology ACL gotcha)
 Synology's `/volume1/pbs` share was initially created with "Advanced
 Permissions" which imposes a `syno_acl` that denies writes to
 non-admin UIDs (even the `backup` uid PBS runs as). Fix was to flatten
 the share to pure POSIX mode 777 so the mount becomes "Linux mode"
 (`synoacltool -get` confirms). `no_root_squash + no_all_squash` also
 set on the NFS export.
 Mount uses NFSv3 (same pattern as pbs-ana, for consistency more than
 necessity — Synology's Btrfs doesn't have the ZFS-case-insensitivity
 issue):
 ```
 10.100.50.50:/volume1/pbs /mnt/pbs-datastore nfs \
  defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=3 0 0
 ```
 Full history: `docs/runbooks/pbs-deployment.md` Phase 5.3.
 ## What runs here
 - `proxmox-backup-proxy` — HTTPS API + web UI (port 8007)
 - `proxmox-backup` — backend service
 - Scheduled verify job: weekly Sunday 12:00 (after nightly sync
  completes), `ignore-verified=true`, `outdated-after=30 days`
 - Scheduled sync from pbs-ana: daily at 06:00 UTC
 ## Remote config
 - Remote name: `pbs-ana`
 - Remote host: `10.250.50.90:8007`
 - Auth: `root@pam!fleet-vzdump` token (same token as clients use on
  the primary — shared here for simplicity; could be split to a
  sync-only read-only token)
 - Cert fingerprint: in password manager under "pbs-ana fingerprint"
 ## Namespaces
 Namespaces are auto-replicated from pbs-ana via the sync job — you do
 NOT need to pre-create them on the mirror. Expected namespaces after
 first sync: same 5 as pbs-ana (`pfi-pve`, `nh3-pve`, `esh-pve`,
 `esh-pve-nas`, `sfsrv-ana`).
 ## Dependencies / blast radius
 - **Hard dependency on nh3-nas** (Synology). Datastore goes dead if
  the Synology crashes; sync fails, verify fails, restore ops fail.
  But: ANA primary keeps operating fine, so the fleet's backup SLA
  is preserved — only DR redundancy is temporarily lost.
 - **Sync source**: pbs-ana going offline means no new snapshots pull
  here, but existing ones remain restorable. Sync resumes
  automatically once pbs-ana recovers.
 ## DR failover (hypothetical)
 If pbs-ana is permanently lost and we needed to restore fleet VMs
 from pbs-nh3:
 1. Re-point each PVE host's `pbs-ana` storage entry at pbs-nh3
   (`pve-datacenter.cfg` → change `server` to 10.100.50.90)
 2. Fingerprint + token still work (same token)
 3. Cross-WAN restore is slower than local but functional
 Documented as future work — not tested. Run a trial restore of a
 small CT periodically to validate.
 ## Refresh
 ```
 scripts/refresh-server-info.sh pbs-nh3
 ```
 Snapshot at `system-details.txt`. Full deployment history:
 `docs/runbooks/pbs-deployment.md`.
@@ -0,0 +1 @@
 lkraven@10.100.50.90
@@ -0,0 +1,84 @@
 ===== HOST =====
 Hostname:   pbs-nh3.phasefinal.com
 Date:       2026-04-23T21:56:13-07:00
 Uptime:     up 22 hours, 41 minutes
 OS:         Debian GNU/Linux 12 (bookworm)
 Kernel:     6.1.0-44-amd64
 Arch:       x86_64
 ===== HARDWARE =====
 CPU cores:  4
 CPU model:  QEMU Virtual CPU version 2.5+
 MemTotal:   7.8 GB
 MemAvailable: 7.3 GB
 ===== GPUS =====
 nvidia-smi not present (no NVIDIA GPUs or driver not installed)
 ===== FILESYSTEMS (df) =====
 Filesystem                 Size  Used Avail Use% Mounted on
 /dev/sda1                   31G  2.0G   27G   7% /
 10.100.50.50:/volume1/pbs   42T   27T   16T  64% /mnt/pbs-datastore
 ===== PERSISTENT MOUNTS (/etc/fstab, non-comment) =====
 UUID=34a61420-a414-4789-8a3d-065a8905eb11 /               ext4    errors=remount-ro 0       1
 UUID=518afc03-e17d-4e1a-a5d4-84b96bee1c60 none            swap    sw              0       0
 /dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
 10.100.50.50:/volume1/pbs  /mnt/pbs-datastore  nfs  defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=3  0  0
 ===== TARGETED DATA PATHS =====
 /opt  (total: 4.0K)
  total 8
  drwxr-xr-x  2 root root 4096 2026-04-22 21:04 .
  drwxr-xr-x 18 root root 4096 2026-04-22 21:06 ..
 /srv  (total: 4.0K)
  total 8
  drwxr-xr-x  2 root root 4096 2026-04-22 21:04 .
  drwxr-xr-x 18 root root 4096 2026-04-22 21:06 ..
 ===== DOCKER =====
 docker not installed
 ===== COMPOSE FILES (/opt/docker/compose/) =====
 /opt/docker/compose not present
 ===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) =====
 /opt/docker/conf not present
 ===== LISTENING PORTS =====
 0.0.0.0:111
 0.0.0.0:22
 0.0.0.0:41669
 0.0.0.0:46653
 [::]:111
 [::1]:25
 127.0.0.1:25
 127.0.0.1:82
 [::]:22
 [::]:37707
 [::]:54621
 *:8007
 ===== MODEL / HUGGINGFACE CACHES =====
 ===== DOCKER-ADJACENT SYSTEMD SERVICES =====
 (none matching)
 ===== DONE =====
 Paste the above back into the chat, or pass a path as argv[1] to save.
@@ -0,0 +1,104 @@
 ===== HOST =====
 Hostname:   pfi-ana-webhost.phasefinal.com
 Date:       2026-04-23T01:00:53-07:00
 Uptime:     up 34 weeks, 4 days, 11 hours, 35 minutes
 OS:         Debian GNU/Linux 11 (bullseye)
 Kernel:     5.10.0-34-amd64
 Arch:       x86_64
 ===== HARDWARE =====
 CPU cores:  16
 CPU model:  Common KVM processor
 MemTotal:   0.8 GB
 MemAvailable: 0.3 GB
 ===== GPUS =====
 nvidia-smi not present (no NVIDIA GPUs or driver not installed)
 ===== FILESYSTEMS (df) =====
 Filesystem      Size  Used Avail Use% Mounted on
 /dev/sda1       245G  8.6G  224G   4% /
 ===== PERSISTENT MOUNTS (/etc/fstab, non-comment) =====
 UUID=110a7e2b-aaf8-4a55-a064-408b665ee322	/	ext4	rw,grpquota,errors=remount-ro,relatime,quota,usrquota	0	1
 UUID=f788f710-a3ec-457e-8053-2eeca8c3f029 none            swap    sw              0       0
 /dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
 /home/dragonfirepro	/home/chroot/169559808468512/home/dragonfirepro	bind	defaults,bind	0	
 ===== TARGETED DATA PATHS =====
 /opt  (total: 4.0K)
  total 8
  drwxr-xr-x  2 root root 4096 2023-09-24 15:42 .
  drwxr-xr-x 18 root root 4096 2025-04-12 13:58 ..
 /srv  (total: 12K)
  total 12
  drwxr-xr-x  3 root root    4096 2023-09-24 16:15 .
  drwxr-xr-x 18 root root    4096 2025-04-12 13:58 ..
  drwxr-xr-x  2 ftp  nogroup 4096 2023-09-24 16:15 ftp
 ===== DOCKER =====
 docker not installed
 ===== COMPOSE FILES (/opt/docker/compose/) =====
 /opt/docker/compose not present
 ===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) =====
 /opt/docker/conf not present
 ===== LISTENING PORTS =====
 0.0.0.0:10000
 0.0.0.0:110
 0.0.0.0:143
 0.0.0.0:20000
 0.0.0.0:22
 0.0.0.0:993
 0.0.0.0:995
 [::]:10000
 10.250.50.52:53
 [::]:110
 [::1]:10023
 [::1]:25
 127.0.0.1:10023
 127.0.0.1:25
 127.0.0.1:3306
 127.0.0.1:465
 127.0.0.1:53
 127.0.0.1:587
 127.0.0.1:953
 [::]:143
 [::1]:465
 [::1]:53
 [::1]:587
 [::1]:953
 *:21
 [::]:22
 *:2222
 *:443
 *:80
 [::]:993
 [::]:995
 [fe80::e4f9:3aff:fec9:612a]%ens18:53
 ===== MODEL / HUGGINGFACE CACHES =====
 ===== DOCKER-ADJACENT SYSTEMD SERVICES =====
 (none matching)
 ===== DONE =====
 Paste the above back into the chat, or pass a path as argv[1] to save.
@@ -0,0 +1,80 @@
 ===== HOST =====
 Hostname:   pfi-postgres.phasefinal.com
 Date:       2026-04-23T01:01:01-07:00
 Uptime:     up 34 weeks, 4 days, 5 hours, 12 minutes
 OS:         Debian GNU/Linux 12 (bookworm)
 Kernel:     6.1.0-38-amd64
 Arch:       x86_64
 ===== HARDWARE =====
 CPU cores:  16
 CPU model:  Common KVM processor
 MemTotal:   7.8 GB
 MemAvailable: 7.2 GB
 ===== GPUS =====
 nvidia-smi not present (no NVIDIA GPUs or driver not installed)
 ===== FILESYSTEMS (df) =====
 Filesystem            Size  Used Avail Use% Mounted on
 /dev/sda1              78G  4.2G   70G   6% /
 10.250.50.50:/mnt/db   20T  281M   20T   1% /mnt/db
 ===== PERSISTENT MOUNTS (/etc/fstab, non-comment) =====
 UUID=e4f13bda-0320-43e7-be36-8df370f5c099 /               ext4    errors=remount-ro 0       1
 UUID=f9efdef0-743c-46b9-8131-ae0a3e07e40a none            swap    sw              0       0
 /dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
 10.250.50.50:/mnt/db /mnt/db nfs defaults 0 0
 ===== TARGETED DATA PATHS =====
 /opt  (total: 4.0K)
  total 8
  drwxr-xr-x  2 root root 4096 2023-04-30 10:11 .
  drwxr-xr-x 18 root root 4096 2025-08-23 19:46 ..
 /srv  (total: 4.0K)
  total 8
  drwxr-xr-x  2 root root 4096 2023-04-30 10:11 .
  drwxr-xr-x 18 root root 4096 2025-08-23 19:46 ..
 ===== DOCKER =====
 docker not installed
 ===== COMPOSE FILES (/opt/docker/compose/) =====
 /opt/docker/compose not present
 ===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) =====
 /opt/docker/conf not present
 ===== LISTENING PORTS =====
 0.0.0.0:111
 0.0.0.0:22
 0.0.0.0:5432
 [::]:111
 127.0.0.1:5433
 [::1]:5433
 [::]:22
 [::]:5432
 ===== MODEL / HUGGINGFACE CACHES =====
 ===== DOCKER-ADJACENT SYSTEMD SERVICES =====
 (none matching)
 ===== DONE =====
 Paste the above back into the chat, or pass a path as argv[1] to save.
@@ -0,0 +1,100 @@
 ===== HOST =====
 Hostname:   pfi-pteradactyl.phasefinal.com
 Date:       2026-04-23T01:00:55-07:00
 Uptime:     up 34 weeks, 4 days, 11 hours, 33 minutes
 OS:         Debian GNU/Linux 12 (bookworm)
 Kernel:     6.1.0-34-amd64
 Arch:       x86_64
 ===== HARDWARE =====
 CPU cores:  8
 CPU model:  Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz
 MemTotal:   7.8 GB
 MemAvailable: 0.3 GB
 ===== GPUS =====
 nvidia-smi not present (no NVIDIA GPUs or driver not installed)
 ===== FILESYSTEMS (df) =====
 Filesystem      Size  Used Avail Use% Mounted on
 /dev/sda1       250G   67G  172G  28% /
 ===== PERSISTENT MOUNTS (/etc/fstab, non-comment) =====
 UUID=028d611a-2cbc-4523-b1f3-2aa04d74302e /               ext4    errors=remount-ro 0       1
 UUID=578e5418-a021-407d-8162-4b725807ee13 none            swap    sw              0       0
 /dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
 ===== TARGETED DATA PATHS =====
 /opt  (total: 4.0K)
  total 8
  drwxr-xr-x  2 root root 4096 2023-05-14 13:22 .
  drwxr-xr-x 18 root root 4096 2025-04-28 17:36 ..
 /var/lib/docker  (total: 4.0K)
 /srv  (total: 4.0K)
  total 8
  drwxr-xr-x  2 root root 4096 2023-05-14 13:22 .
  drwxr-xr-x 18 root root 4096 2025-04-28 17:36 ..
 ===== DOCKER =====
 Server: 
 docker daemon not reachable by current user
 ----- docker info -----
 Containers:      
 ----- running containers -----
 ----- all containers -----
 ----- networks -----
 ----- networks (external, non-default — worth knowing for compose external: true) -----
 ----- named volumes -----
 ----- compose projects currently running -----
 (none)
 ===== COMPOSE FILES (/opt/docker/compose/) =====
 /opt/docker/compose not present
 ===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) =====
 /opt/docker/conf not present
 ===== LISTENING PORTS =====
 0.0.0.0:22
 0.0.0.0:80
 10.250.50.55:7777
 127.0.0.1:3306
 127.0.0.1:34135
 127.0.0.1:6379
 [::1]:6379
 *:2022
 [::]:22
 *:8080
 ===== MODEL / HUGGINGFACE CACHES =====
 ===== DOCKER-ADJACENT SYSTEMD SERVICES =====
 containerd.service running
 docker.service running
 ===== DONE =====
 Paste the above back into the chat, or pass a path as argv[1] to save.
@@ -4,13 +4,16 @@ Proxmox VE hypervisor for the Anaheim colo (`pve.phasefinal.com`). Hosts every V
 ## Network
- **LAN IP:** 10.250.250.31
+- **LAN IP:** 10.250.250.31 (in-band, OS-side)
 - **iDRAC (OOB):** 10.250.250.30 — Dell iDRAC web UI at
  <https://10.250.250.30/> (homepage card: *PFI-r750xs-iDRAC*)
 - **FQDN:** `pve.phasefinal.com`
 - **Web UI:** `https://10.250.250.31:8006`
 - **SSH:** `root@pfi-pve` (key auth; `ssh-target` file uses the hostname)
 ## Hardware
 - **Chassis:** Dell PowerEdge R750xs (2U rackmount)
 - **CPU:** Intel Xeon Silver 4310 @ 2.10 GHz
 - **RAM:** 188 GB
 - **Kernel:** `6.8.12-8-pve` (Proxmox 8.x)
@@ -18,9 +21,24 @@ Proxmox VE hypervisor for the Anaheim colo (`pve.phasefinal.com`). Hosts every V
 ## What it runs
-The VMs in [`docs/pfi/proxmox-vms.md`](../../docs/pfi/proxmox-vms.md) live here — VM 100 (PFI-ANA-TRUENAS, now Debian NFS/SMB), VM 101 (DC), VM 102 (PFI-ANA-Docker = `ana-docker`), 103–111 (Mongo, Postgres, Tailscale, Pteradactyl, Webhost, tacticalrmm, etc.).
+The VMs/CTs in [`docs/pfi/proxmox-vms.md`](../../docs/pfi/proxmox-vms.md)
 live here. Known as of 2026-04-23 (from vzdump logs + `servers/`
 dirs; not exhaustive):
-`ana-ml2` (`10.250.50.54`) is a separate bare-metal box, not on this hypervisor.
+- **VM 102** — `ana-docker` (PFI-ANA-Docker)
 - **VM 105** — `pfi-postgres` (PFI-Postgres; PGDATA on NFS from ana-nas)
 - **VM 107** — `pfi-pteradactyl` (game panel)
 - **VM 110** — `pfi-ana-webhost`
 - **VM 111** — `pfi-tacticalrmm`
 - **VM (TBD)** — `pbs-ana` (Proxmox Backup Server; 10.250.50.90)
 - **CT 109** — `ana-nas` (Debian LXC serving NFS/SMB; bind-mounts
  from ospool; data-serving SPOF — see `memory/incident_ana_nas_spof.md`)
 - **CT 112** — `ana-filebot` (file-task automation)
 - **CT 113** — `ana-wg` (WireGuard VPN)
 `ana-ml2` (`10.250.50.54`) is a separate bare-metal Supermicro box,
 not on this hypervisor. `sfsrv-ana` (`10.250.250.115`) runs on a
 separate Dell R630, also not on this hypervisor.
 ## Refresh state
@@ -0,0 +1,87 @@
 ===== HOST =====
 Hostname:   pfi-tacticalrmm.phasefinal.com
 Date:       2026-04-23T01:01:00-07:00
 Uptime:     up 2 days, 6 hours, 0 minutes
 OS:         Debian GNU/Linux 12 (bookworm)
 Kernel:     6.1.0-32-amd64
 Arch:       x86_64
 ===== HARDWARE =====
 CPU cores:  16
 CPU model:  Common KVM processor
 MemTotal:   7.8 GB
 MemAvailable: 6.0 GB
 ===== GPUS =====
 nvidia-smi not present (no NVIDIA GPUs or driver not installed)
 ===== FILESYSTEMS (df) =====
 Filesystem      Size  Used Avail Use% Mounted on
 /dev/sda1       250G  7.1G  231G   3% /
 ===== PERSISTENT MOUNTS (/etc/fstab, non-comment) =====
 UUID=ce4c34ea-d1fa-46ea-90c9-5538c1a077c4 /               ext4    errors=remount-ro 0       1
 UUID=125637a3-f2eb-492f-85ad-96c4bbda1387 none            swap    sw              0       0
 /dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
 ===== TARGETED DATA PATHS =====
 /opt  (total: 9.8M)
  total 16
  drwxr-xr-x  4 root     root     4096 2023-11-02 12:34 .
  drwxr-xr-x 20 root     root     4096 2025-04-07 14:39 ..
  drwxr-xr-x  3 tactical tactical 4096 2023-11-02 12:34 tactical
  drwxr-xr-x  8 tactical tactical 4096 2026-04-20 20:37 trmm-community-scripts
 /srv  (total: 4.0K)
  total 8
  drwxr-xr-x  2 root root 4096 2023-11-02 11:18 .
  drwxr-xr-x 20 root root 4096 2025-04-07 14:39 ..
 ===== DOCKER =====
 docker not installed
 ===== COMPOSE FILES (/opt/docker/compose/) =====
 /opt/docker/compose not present
 ===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) =====
 /opt/docker/conf not present
 ===== LISTENING PORTS =====
 0.0.0.0:22
 0.0.0.0:443
 0.0.0.0:80
 *:1024
 127.0.0.1:4222
 127.0.0.1:5432
 127.0.0.1:6379
 127.0.0.1:9235
 [::1]:5432
 [::1]:6379
 [::]:22
 [::]:443
 *:4430
 *:4433
 [::]:80
 ===== MODEL / HUGGINGFACE CACHES =====
 ===== DOCKER-ADJACENT SYSTEMD SERVICES =====
 (none matching)
 ===== DONE =====
 Paste the above back into the chat, or pass a path as argv[1] to save.
@@ -0,0 +1,249 @@
 ===== HOST =====
 Hostname:   sf-ana-container.surefire.com
 Date:       2026-04-21T16:57:55-07:00
 Uptime:     up 3 weeks, 6 days, 1 hour, 17 minutes
 OS:         Debian GNU/Linux 12 (bookworm)
 Kernel:     6.1.0-21-amd64
 Arch:       x86_64
 ===== HARDWARE =====
 CPU cores:  16
 CPU model:  QEMU Virtual CPU version 2.5+
 MemTotal:   15.6 GB
 MemAvailable: 14.4 GB
 ===== GPUS =====
 nvidia-smi not present (no NVIDIA GPUs or driver not installed)
 ===== FILESYSTEMS (df) =====
 Filesystem      Size  Used Avail Use% Mounted on
 /dev/sda1       250G  7.1G  231G   3% /
 ===== PERSISTENT MOUNTS (/etc/fstab, non-comment) =====
 UUID=fcc00570-1585-41d4-8c64-8e5d59af40e0 /               ext4    errors=remount-ro 0       1
 UUID=35ad80f9-5952-489a-9178-575be588420c none            swap    sw              0       0
 /dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
 ===== TARGETED DATA PATHS =====
 /opt  (total: 8.3M)
  total 12
  drwxr-xr-x  3 root root 4096 2024-06-07 10:42 .
  drwxr-xr-x 18 root root 4096 2024-06-07 10:35 ..
  drwxrwxrwx  5 root root 4096 2024-06-07 10:46 docker
 /opt/docker  (total: 8.3M)
  total 20
  drwxrwxrwx 5 root   root   4096 2024-06-07 10:46 .
  drwxr-xr-x 3 root   root   4096 2024-06-07 10:42 ..
  drwxr-xr-x 6 vhoang vhoang 4096 2024-06-07 11:09 compose
  drwxr-xr-x 3 vhoang vhoang 4096 2024-06-07 10:46 conf
  drwxr-xr-x 2 vhoang vhoang 4096 2024-06-07 10:42 data
 /opt/docker/compose  (total: 8.3M)
  total 24
  drwxr-xr-x 6 vhoang vhoang 4096 2024-06-07 11:09 .
  drwxrwxrwx 5 root   root   4096 2024-06-07 10:46 ..
  drwxr-xr-x 2 vhoang vhoang 4096 2024-06-07 10:43 dockge
  drwxr-xr-x 4 root   root   4096 2024-06-07 10:58 nginx-proxy
  drwxr-xr-x 2 root   root   4096 2024-06-07 10:46 traefik
  drwxr-xr-x 3 root   root   4096 2024-06-07 11:09 vaultwarden
 /opt/docker/conf  (total: 8.0K)
  total 12
  drwxr-xr-x 3 vhoang vhoang 4096 2024-06-07 10:46 .
  drwxrwxrwx 5 root   root   4096 2024-06-07 10:46 ..
  drwxr-xr-x 2 root   root   4096 2024-06-07 10:46 traefik-esh
 /var/lib/docker  (total: 4.0K)
 /srv  (total: 4.0K)
  total 8
  drwxr-xr-x  2 root root 4096 2023-11-09 17:45 .
  drwxr-xr-x 18 root root 4096 2024-06-07 10:35 ..
 ===== DOCKER =====
 Server: 
 docker daemon not reachable by current user
 ----- docker info -----
 Containers:      
 ----- running containers -----
 ----- all containers -----
 ----- networks -----
 ----- networks (external, non-default — worth knowing for compose external: true) -----
 ----- named volumes -----
 ----- compose projects currently running -----
 (none)
 ===== COMPOSE FILES (/opt/docker/compose/) =====
 >>> /opt/docker/compose/dockge/compose.yaml
 services:
  dockge:
    image: louislam/dockge:latest
    restart: unless-stopped
    ports:
      # Host Port : Container Port
      - 5001:5001
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - dockge_data:/app/data
      - /opt/docker/compose:/opt/docker/compose
    environment:
      # Tell Dockge where is your stacks directory
      - DOCKGE_STACKS_DIR=/opt/docker/compose
    networks:
      - tnet
 volumes:
  dockge_data: null
 networks:
  tnet:
    name: traefik-net
    external: true
 >>> /opt/docker/compose/nginx-proxy/compose.yaml
 version: "3.8"
 services:
  app:
    image: jc21/nginx-proxy-manager:latest
    restart: unless-stopped
    ports:
      # These ports are in format <host-port>:<container-port>
      - 80:80 # Public HTTP Port
      - 443:443 # Public HTTPS Port
      - 81:81 # Admin Web Port
      # Add any other Stream port you want to expose
      # - '21:21' # FTP
      # Uncomment the next line if you uncomment anything in the section
      # environment:
      # Uncomment this if you want to change the location of
      # the SQLite DB file within the container
      # DB_SQLITE_FILE: "/data/database.sqlite"
      # Uncomment this if IPv6 is not enabled on your host
      # DISABLE_IPV6: 'true'
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
 networks:
  default:
    external: true
    name: nginx-proxy_default
 >>> /opt/docker/compose/traefik/compose.yaml
 #version: "3.3"
 services:
  traefik:
    image: traefik:latest
    container_name: traefik
    command:
      - --log.level=DEBUG
      - --configFile=/etc/traefik/traefik.yml
      - --api.insecure=true
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --entrypoints.web.address=:80
    environment:
      - AWS_ACCESS_KEY_ID=${AWS_KEY}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET}
      - AWS_REGION=${AWS_REGION}
      - AWS_HOSTED_ZONE_ID=${AWS_ZONEID}
    ports:
      - 80:80
      - 8380:8080
      - 443:443
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /opt/docker/conf/traefik-esh:/etc/traefik
    networks:
      - tnet
    env_file:
      - .env
 networks:
  tnet:
    name: traefik-net
    external: true
 >>> /opt/docker/compose/vaultwarden/compose.yaml
 version: "3"
 services:
  vaultwarden:
    image: vaultwarden/server:latest
    container_name: vaultwarden
    restart: unless-stopped
    ports:
      - 9445:80 #map any custom port to use (replace 8445 not 80)
    volumes:
      - ./bitwarden:/data:rw
    environment:
      - ADMIN_TOKEN=${ADMIN_TOKEN}
      - WEBSOCKET_ENABLED=true
      - SIGNUPS_ALLOWED=true
      - DOMAIN=${DOMAIN}
      - SMTP_HOST=surefire-com.mail.protection.outlook.com
      - SMTP_FROM=vaultwarden@surefire.com
      - SMTP_PORT=25
 #      - SMTP_USERNAME=relay@surefire.com
 #      - SMTP_PASSWORD=5Gb&237N@p%6
 networks:
  default:
    external: true
    name: nginx-proxy_default
 ===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) =====
 /opt/docker/conf
 /opt/docker/conf/traefik-esh
 ===== LISTENING PORTS =====
 0.0.0.0:22
 0.0.0.0:443
 0.0.0.0:5001
 0.0.0.0:80
 0.0.0.0:8000
 0.0.0.0:81
 0.0.0.0:9443
 0.0.0.0:9445
 127.0.0.1:46567
 [::]:22
 [::]:443
 [::]:5001
 [::]:80
 [::]:8000
 [::]:81
 [::]:9443
 [::]:9445
 ===== MODEL / HUGGINGFACE CACHES =====
 ===== DOCKER-ADJACENT SYSTEMD SERVICES =====
 containerd.service running
 docker.service running
 ===== DONE =====
 Paste the above back into the chat, or pass a path as argv[1] to save.
@@ -3,43 +3,52 @@
 SureFire-client physical server at the Anaheim colo — Dell PowerEdge
 R630. **PFI-managed** under the hosting agreement (hardware + OS ops).
 **This directory documents the hardware / iDRAC side only.** The OS
 running on this box is tracked separately as
 [`servers/sfsrv-ana/`](../sfsrv-ana/README.md) — Proxmox VE at
 `10.250.250.115`. Refer to that for OS-level snapshots, SSH access,
 and backup coverage.
 ## Client context
 - **Client:** SureFire
- **PFI role:** full-service managed host
+- **PFI role:** full-service managed host (iDRAC + Proxmox)
 ## Network
- **iDRAC BMC IP:** 10.250.250.110 (on the management subnet)
+- **iDRAC BMC:** `10.250.250.110` (dedicated OOB management NIC)
-  - Web UI: https://10.250.250.110/
+  - Web UI: <https://10.250.250.110/> (currently 404s; use the
- **OS-side LAN IP:** **unknown** — not in FortiGate DHCP (static
+    vSphere-style Redfish API or racadm from iDRAC shell for
-  config?). Fill this in once identified.
+    out-of-band ops)
- **SSH:** no `ssh-target` set yet because the OS IP isn't known. Add
+  - Homepage monitors this via ICMP ping rather than HTTP
-  one (e.g. `root@<ip>` or `lkraven@<ip>`) once identified.
+- **OS-side (Proxmox VE):** see `servers/sfsrv-ana/` — `10.250.250.115`
-## Infrastructure
+## Hardware
- **Type:** Physical Dell PowerEdge R630
+- **Type:** Physical Dell PowerEdge R630, 2U
 - **Site:** Anaheim (PFI colo)
 - Detailed hardware / CPU / RAM / disk data comes from `sfsrv-ana`'s
  `system-details.txt` (the OS can read all of this via `dmidecode`,
  `lscpu`, etc. — no need to duplicate here).
-## Backup coverage
+## Why this dir exists separately from sfsrv-ana
- **VM-image:** N/A (bare-metal physical)
+- The iDRAC is a distinct management surface with its own credentials,
- **File-level restic:** not yet configured
+  protocol stack, and firmware lifecycle. Power events, BIOS updates,
-  - Pending OS identification + SSH access
+  and console access live here.
-  - Follow the `configs/restic/ana-ml2/` template when setting up
+- Inventory audits that track *hardware* (for hosting-agreement
-    (both are bare-metal hosts; similar source list + no DB hooks by
+  billing, depreciation, etc.) should see the physical box as its own
-    default unless SF apps run databases)
+  entry independent of whatever OS happens to be running on it.
 ## Refresh state
-Once the OS IP is known and an `ssh-target` is in place:
+**No `ssh-target` and no `system-details.txt`** — iDRAC doesn't expose
-```bash
+a useful sshd for our inspect scripts. Host metadata comes from
-scripts/refresh-server-info.sh sf-r630
+`sfsrv-ana`'s snapshot instead.
 ```
 ## Discovered via
 `scripts/discover-fortigate.sh 10.250.250.1` on 2026-04-21 — iDRAC's
 DHCP lease on the management interface (MAC `74:e6:e2:fe:2c:7c`,
-VCI `iDRAC`).
+VCI `iDRAC`). Matched to sfsrv-ana via iDRAC Service Tag / physical
 co-location on 2026-04-23.
@@ -0,0 +1,164 @@
 ===== HOST =====
 Hostname:   sfsrv-ana.surefire.com
 Date:       2026-04-21T16:57:44-07:00
 Uptime:     up 3 weeks, 6 days, 1 hour, 17 minutes
 PVE:        pve-manager/8.2.2/9355359cd7afbae4 (running kernel: 6.8.4-3-pve)
 Kernel:     6.8.4-3-pve
 Arch:       x86_64
 ===== HARDWARE =====
 CPU model:  Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
 CPU cores:  24
 MemTotal:   110.0 GB
 MemAvail:   104.1 GB
 ===== CLUSTER =====
 (single-node host (or pvecm returned no cluster info))
 ===== CLUSTER RESOURCES (pvesh) =====
 ┌─────────────────────────────┬─────────┬─────────────┬───────────────────┬───────┬───────────┬─────────┬───────┬────────┬────────────┬────────────┬──────────┬──────────────────┬───────────┬────────────┬──────┬───────────┬───────────┬──────────┬──────┐
 │ id                          │ type    │ cgroup-mode │ content           │   cpu │      disk │ hastate │ level │ maxcpu │    maxdisk │     maxmem │      mem │ name             │ node      │ plugintype │ pool │ status    │ storage   │   uptime │ vmid │
 ╞═════════════════════════════╪═════════╪═════════════╪═══════════════════╪═══════╪═══════════╪═════════╪═══════╪════════╪════════════╪════════════╪══════════╪══════════════════╪═══════════╪════════════╪══════╪═══════════╪═══════════╪══════════╪══════╡
 │ node/sfsrv-ana              │ node    │           2 │                   │ 0.32% │  6.71 GiB │         │       │     24 │  93.93 GiB │ 110.05 GiB │ 5.05 GiB │                  │ sfsrv-ana │            │      │ online    │           │ 3w 6d 1h │      │
 ├─────────────────────────────┼─────────┼─────────────┼───────────────────┼───────┼───────────┼─────────┼───────┼────────┼────────────┼────────────┼──────────┼──────────────────┼───────────┼────────────┼──────┼───────────┼───────────┼──────────┼──────┤
 │ qemu/100                    │ qemu    │             │                   │ 0.33% │    0.00 B │         │       │     16 │ 256.00 GiB │  16.00 GiB │ 2.99 GiB │ sf-ana-container │ sfsrv-ana │            │      │ running   │           │ 3w 6d 1h │  100 │
 ├─────────────────────────────┼─────────┼─────────────┼───────────────────┼───────┼───────────┼─────────┼───────┼────────┼────────────┼────────────┼──────────┼──────────────────┼───────────┼────────────┼──────┼───────────┼───────────┼──────────┼──────┤
 │ sdn/sfsrv-ana/localnetwork  │ sdn     │             │                   │       │           │         │       │        │            │            │          │                  │ sfsrv-ana │            │      │ ok        │           │          │      │
 ├─────────────────────────────┼─────────┼─────────────┼───────────────────┼───────┼───────────┼─────────┼───────┼────────┼────────────┼────────────┼──────────┼──────────────────┼───────────┼────────────┼──────┼───────────┼───────────┼──────────┼──────┤
 │ storage/sfsrv-ana/local     │ storage │             │ backup,iso,vztmpl │       │  6.71 GiB │         │       │        │  93.93 GiB │            │          │                  │ sfsrv-ana │ dir        │      │ available │ local     │          │      │
 ├─────────────────────────────┼─────────┼─────────────┼───────────────────┼───────┼───────────┼─────────┼───────┼────────┼────────────┼────────────┼──────────┼──────────────────┼───────────┼────────────┼──────┼───────────┼───────────┼──────────┼──────┤
 │ storage/sfsrv-ana/local-lvm │ storage │             │ rootdir,images    │       │ 13.97 GiB │         │       │        │ 793.79 GiB │            │          │                  │ sfsrv-ana │ lvmthin    │      │ available │ local-lvm │          │      │
 └─────────────────────────────┴─────────┴─────────────┴───────────────────┴───────┴───────────┴─────────┴───────┴────────┴────────────┴────────────┴──────────┴──────────────────┴───────────┴────────────┴──────┴───────────┴───────────┴──────────┴──────┘
 ===== VMs (qm list) =====
      VMID NAME                 STATUS     MEM(MB)    BOOTDISK(GB) PID       
       100 sf-ana-container     running    16384            256.00 1361      
 ===== VM CONFIGS (qm config per id) =====
 ----- VMID 100 -----
 agent: 1
 balloon: 0
 boot: order=scsi0;ide2;net0
 cores: 4
 ide2: local:iso/debian-12.2.0-amd64-netinst.iso,media=cdrom,size=628M
 memory: 16384
 name: sf-ana-container
 net0: virtio=6A:09:9A:8A:08:A5,bridge=vmbr0,firewall=1,tag=150
 onboot: 1
 ostype: l26
 scsi0: local-lvm:vm-100-disk-0,iothread=1,size=256G
 ===== LXC CONTAINERS (pct list) =====
 ===== LXC CONFIGS (pct config per id) =====
 (no LXC containers found)
 ===== STORAGE (pvesm status) =====
 Name             Type     Status           Total            Used       Available        %
 local             dir     active        98497780         7034220        86414012    7.14%
 local-lvm     lvmthin     active       832352256        14649399       817702856    1.76%
 ===== DATASTORE CONFIG (/etc/pve/storage.cfg) =====
 dir: local
 	path /var/lib/vz
 	content iso,vztmpl,backup
 lvmthin: local-lvm
 	thinpool data
 	vgname pve
 	content rootdir,images
 ===== ZFS POOLS (zpool list -v) =====
 no pools available
 ===== FILESYSTEMS (df -h, local fs only) =====
 Filesystem           Type  Size  Used Avail Use% Mounted on
 /dev/mapper/pve-root ext4   94G  6.8G   83G   8% /
 /dev/fuse            fuse  128M   24K  128M   1% /etc/pve
 ===== MOUNTED NFS / CIFS =====
 ===== BACKUP JOBS (/etc/pve/jobs.cfg + /etc/vzdump.conf) =====
 (/etc/pve/jobs.cfg not readable (no scheduled backups?))
 ----- vzdump defaults -----
 ===== BACKUP COVERAGE ANALYSIS =====
 (/etc/pve/jobs.cfg not readable — no coverage data)
 ----- per-guest coverage -----
 ID      VERDICT     REASON
 100     NO          no job targets this id
 ===== RECENT BACKUPS ON LOCAL DUMP STORAGES =====
 ----- /var/lib/vz/dump/ -----
 ===== PBS TARGETS =====
 (no PBS storage configured on this node)
 ===== REPLICATION JOBS (pvesr status) =====
 JobID      Enabled    Target                           LastSync             NextSync   Duration  FailCount State
 ===== LISTENING PORTS =====
 0.0.0.0:111
 0.0.0.0:22
 [::]:111
 [::1]:25
 127.0.0.1:25
 127.0.0.1:85
 [::]:22
 *:3128
 *:8006
 ===== SUBSCRIPTION / UPDATES =====
 status: notfound
 ----- apt upgradable (top 20) -----
 base-files/oldstable 12.4+deb12u13 amd64 [upgradable from: 12.4+deb12u5]
 bash/oldstable 5.2.15-2+b10 amd64 [upgradable from: 5.2.15-2+b2]
 bind9-dnsutils/oldstable-security 1:9.18.47-1~deb12u1 amd64 [upgradable from: 1:9.18.24-1]
 bind9-host/oldstable-security 1:9.18.47-1~deb12u1 amd64 [upgradable from: 1:9.18.24-1]
 bind9-libs/oldstable-security 1:9.18.47-1~deb12u1 amd64 [upgradable from: 1:9.18.24-1]
 bsdextrautils/oldstable 2.38.1-5+deb12u3 amd64 [upgradable from: 2.38.1-5+deb12u1]
 bsdutils/oldstable 1:2.38.1-5+deb12u3 amd64 [upgradable from: 1:2.38.1-5+deb12u1]
 btrfs-progs/oldstable 6.2-1+deb12u2 amd64 [upgradable from: 6.2-1]
 busybox/oldstable 1:1.35.0-4+b7 amd64 [upgradable from: 1:1.35.0-4+b3]
 ca-certificates/oldstable,oldstable-updates 20230311+deb12u1 all [upgradable from: 20230311]
 ceph-common/oldstable,oldstable-security 16.2.15+ds-0+deb12u1 amd64 [upgradable from: 16.2.11+ds-2]
 ceph-fuse/oldstable,oldstable-security 16.2.15+ds-0+deb12u1 amd64 [upgradable from: 16.2.11+ds-2]
 corosync/stable 3.1.10-pve2~bpo12+1 amd64 [upgradable from: 3.1.7-pve3]
 criu/oldstable 3.17.1-2+deb12u2 amd64 [upgradable from: 3.17.1-2]
 curl/oldstable 7.88.1-10+deb12u14 amd64 [upgradable from: 7.88.1-10+deb12u5]
 debian-archive-keyring/oldstable 2023.3+deb12u2 all [upgradable from: 2023.3+deb12u1]
 dirmngr/oldstable 2.2.40-1.1+deb12u2 amd64 [upgradable from: 2.2.40-1.1]
 distro-info-data/oldstable 0.58+deb12u6 all [upgradable from: 0.58+deb12u1]
 e2fsprogs/oldstable 1.47.0-2+b2 amd64 [upgradable from: 1.47.0-2]
 eject/oldstable 2.38.1-5+deb12u3 amd64 [upgradable from: 2.38.1-5+deb12u1]
 ===== DONE =====
 Review this output for backup gaps (VERDICT=NO in coverage section)
 and anomalies in storage / PBS / replication sections.
@@ -0,0 +1,80 @@
 ===== HOST =====
 Hostname:   sfsrv-ana.surefire.com
 Date:       2026-04-23T00:52:46-07:00
 Uptime:     up 4 weeks, 9 hours, 12 minutes
 OS:         Debian GNU/Linux 12 (bookworm)
 Kernel:     6.8.4-3-pve
 Arch:       x86_64
 ===== HARDWARE =====
 CPU cores:  24
 CPU model:  Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
 MemTotal:   110.0 GB
 MemAvailable: 104.1 GB
 ===== GPUS =====
 nvidia-smi not present (no NVIDIA GPUs or driver not installed)
 ===== FILESYSTEMS (df) =====
 Filesystem            Size  Used Avail Use% Mounted on
 /dev/mapper/pve-root   94G  6.8G   83G   8% /
 /dev/fuse             128M   24K  128M   1% /etc/pve
 ===== PERSISTENT MOUNTS (/etc/fstab, non-comment) =====
 /dev/pve/root / ext4 errors=remount-ro 0 1
 /dev/pve/swap none swap sw 0 0
 proc /proc proc defaults 0 0
 ===== TARGETED DATA PATHS =====
 /opt  (total: 4.0K)
  total 8
  drwxr-xr-x  2 root root 4096 2023-03-22 07:38 .
  drwxr-xr-x 18 root root 4096 2024-05-08 19:29 ..
 /srv  (total: 4.0K)
  total 8
  drwxr-xr-x  2 root root 4096 2023-03-22 07:38 .
  drwxr-xr-x 18 root root 4096 2024-05-08 19:29 ..
 ===== DOCKER =====
 docker not installed
 ===== COMPOSE FILES (/opt/docker/compose/) =====
 /opt/docker/compose not present
 ===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) =====
 /opt/docker/conf not present
 ===== LISTENING PORTS =====
 0.0.0.0:111
 0.0.0.0:22
 [::]:111
 [::1]:25
 127.0.0.1:25
 127.0.0.1:85
 [::]:22
 *:3128
 *:8006
 ===== MODEL / HUGGINGFACE CACHES =====
 ===== DOCKER-ADJACENT SYSTEMD SERVICES =====
 (none matching)
 ===== DONE =====
 Paste the above back into the chat, or pass a path as argv[1] to save.