vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

servers: add new host dirs, refresh fleet snapshots, orientation doc

Browse Source 
Bundles the inventory expansion since 2026-04-22:

- New host dirs (READMEs + ssh-target where dir name doesn't resolve):
    ana-nas, ana-wg, esh-vm-db, nh3-nas, pbs-ana, pbs-nh3.
- New PFI VM snapshots (registered + key-installed 2026-04-23):
    ana-filebot, pfi-ana-webhost, pfi-postgres, pfi-pteradactyl,
    pfi-tacticalrmm, sf-ana-container, sfsrv-ana (system + proxmox).
- servers/irv-ml1: ONBOARDING.md (the first-time setup notes from when
  the host was brought into the fleet) + ssh-target (10.100.79.3 over
  the WG tunnel — name doesn't DNS-resolve from this workstation).
- servers/{ana-ml2,pfi-pve,sf-r630}/README.md: updates to capture BMC
  IPs, the iDRAC vs OS hostname distinction (sf-r630 hardware =
  sfsrv-ana OS), and the ana-ml2 Supermicro BMC (10.250.250.50,
  distinct from the Dell R750xs iDRAC).
- configs/homepage/docker.yaml: irv-ml1-docker provider added so
  homepage auto-discovers irv-ml1's stacks over the WG tunnel.
- docs/orientation.md: narrative fleet overview written for fresh
  Claude sessions — sites, backup architecture, governing principles,
  gotchas, where-to-look guide. Pointed at from CLAUDE.md.
This commit is contained in:
vh
2026-04-24 21:56:46 -07:00
parent 574c72daa5
commit 60367b307f
31 changed files with 2391 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files
servers/pbs-ana/README.md
+87
View File
@@ -0,0 +1,87 @@
# pbs-ana
Proxmox Backup Server — **fleet primary**. VM on pfi-pve with an
NFS-mounted datastore on the Ana NAS. All 5 hypervisors (pfi-pve,
nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) back up here; one-way sync
to pbs-nh3 at 06:00 daily.
## Network
- **LAN IP:** 10.250.50.90
- **FQDN:** `pbs-ana.phasefinal.com`
- **PBS web UI:** <https://10.250.50.90:8007/>
- **SSH:** `ssh pbs-ana` (config alias → `lkraven@10.250.50.90`).
Root ops via sudo from lkraven.
- **Homepage card:** *PBS-ANA* in `Infra - ANA` group.
## Hardware (VM)
- **Hypervisor:** pfi-pve
- **vCPU:** 4 (QEMU)
- **RAM:** 7.8 GB
- **Root disk:** `/dev/sda1` 31 GB ext4 — OS only (2 GB used)
- **Datastore:** **NFS from ana-nas** at `/mnt/pbs-datastore` →
`10.250.50.50:/mnt/backup/pbs-ana`, 20 TB (currently 336 GB used)
## NFS details (the ZFS-case-insensitivity gotcha)
The ZFS dataset backing `/mnt/backup` on ana-nas is
`casesensitivity=insensitive`, which breaks NFSv4 writes (EACCES).
Mount must be **NFSv3** — that's what `/etc/fstab` already uses:
```
10.250.50.50:/mnt/backup/pbs-ana /mnt/pbs-datastore nfs \
defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=3 0 0
```
Don't upgrade this to vers=4 without fixing the dataset. See
`docs/runbooks/pbs-deployment.md` Phase 0.5 for the full history.
## What runs here
- `proxmox-backup-proxy` — HTTPS API + web UI (port 8007)
- `proxmox-backup` — backend service
- Scheduled verify job: weekly Saturday 23:00 (`ignore-verified=true`,
`outdated-after=30 days`)
- Garbage collection: default weekly
## Namespaces
Per-hypervisor namespaces to prevent VMID collisions across PVE
hosts (VMIDs are NOT globally unique):
- `pfi-pve`
- `nh3-pve`
- `esh-pve`
- `esh-pve-nas`
- `sfsrv-ana`
## API token (for hypervisor clients)
- `root@pam!fleet-vzdump` — stored in password manager; referenced
by each hypervisor's `/etc/pve/storage.cfg` `pbs-ana` stanza.
## Sync to DR
One-way pull: **pbs-nh3** pulls from here at 06:00 daily. pbs-ana
does not push; all coordination is on the DR side.
## Dependencies / blast radius
- **Hard dependency on ana-nas** (CT 109). Datastore NFS goes dead
if ana-nas crashes. Fleet-wide vzdump + verify + sync all fail
until NFS recovers. See `memory/incident_ana_nas_spof.md`.
- **Self-backup excluded** from scheduled vzdump — backing up a
PBS VM to itself is a circular dependency. Config-only backup
could be captured via file-level restic if needed; not currently
set up (low priority — the datastore is on NFS, OS is
reinstallable in ~30 min).
## Refresh
```
scripts/refresh-server-info.sh pbs-ana
```
Snapshot at `system-details.txt`. Full deployment history:
`docs/runbooks/pbs-deployment.md`.
servers/pbs-ana/ssh-target
+1
View File
@@ -0,0 +1 @@
lkraven@10.250.50.90
servers/pbs-ana/system-details.txt
+84
View File
@@ -0,0 +1,84 @@
===== HOST =====
Hostname: pbs-ana.phasefinal.com
Date: 2026-04-23T21:55:34-07:00
Uptime: up 7 hours, 26 minutes
OS: Debian GNU/Linux 12 (bookworm)
Kernel: 6.1.0-44-amd64
Arch: x86_64
===== HARDWARE =====
CPU cores: 4
CPU model: QEMU Virtual CPU version 2.5+
MemTotal: 7.8 GB
MemAvailable: 7.4 GB
===== GPUS =====
nvidia-smi not present (no NVIDIA GPUs or driver not installed)
===== FILESYSTEMS (df) =====
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 31G 2.2G 27G 8% /
10.250.50.50:/mnt/backup/pbs-ana 20T 336G 19T 2% /mnt/pbs-datastore
===== PERSISTENT MOUNTS (/etc/fstab, non-comment) =====
UUID=6a2b24f3-4134-40ed-9b64-f708dcb77cef / ext4 errors=remount-ro 0 1
UUID=bfe22def-87a6-4ca8-8f06-19bbf148559c none swap sw 0 0
/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0
10.250.50.50:/mnt/backup/pbs-ana /mnt/pbs-datastore nfs defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=3 0 0
===== TARGETED DATA PATHS =====
/opt (total: 4.0K)
total 8
drwxr-xr-x 2 root root 4096 2026-04-21 21:22 .
drwxr-xr-x 18 root root 4096 2026-04-21 21:25 ..
/srv (total: 4.0K)
total 8
drwxr-xr-x 2 root root 4096 2026-04-21 21:22 .
drwxr-xr-x 18 root root 4096 2026-04-21 21:25 ..
===== DOCKER =====
docker not installed
===== COMPOSE FILES (/opt/docker/compose/) =====
/opt/docker/compose not present
===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) =====
/opt/docker/conf not present
===== LISTENING PORTS =====
0.0.0.0:111
0.0.0.0:22
0.0.0.0:36203
0.0.0.0:36419
[::]:111
[::1]:25
127.0.0.1:25
127.0.0.1:82
[::]:22
[::]:35545
[::]:56445
*:8007
===== MODEL / HUGGINGFACE CACHES =====
===== DOCKER-ADJACENT SYSTEMD SERVICES =====
(none matching)
===== DONE =====
Paste the above back into the chat, or pass a path as argv[1] to save.