servers: add new host dirs, refresh fleet snapshots, orientation doc
Bundles the inventory expansion since 2026-04-22:
- New host dirs (READMEs + ssh-target where dir name doesn't resolve):
ana-nas, ana-wg, esh-vm-db, nh3-nas, pbs-ana, pbs-nh3.
- New PFI VM snapshots (registered + key-installed 2026-04-23):
ana-filebot, pfi-ana-webhost, pfi-postgres, pfi-pteradactyl,
pfi-tacticalrmm, sf-ana-container, sfsrv-ana (system + proxmox).
- servers/irv-ml1: ONBOARDING.md (the first-time setup notes from when
the host was brought into the fleet) + ssh-target (10.100.79.3 over
the WG tunnel — name doesn't DNS-resolve from this workstation).
- servers/{ana-ml2,pfi-pve,sf-r630}/README.md: updates to capture BMC
IPs, the iDRAC vs OS hostname distinction (sf-r630 hardware =
sfsrv-ana OS), and the ana-ml2 Supermicro BMC (10.250.250.50,
distinct from the Dell R750xs iDRAC).
- configs/homepage/docker.yaml: irv-ml1-docker provider added so
homepage auto-discovers irv-ml1's stacks over the WG tunnel.
- docs/orientation.md: narrative fleet overview written for fresh
Claude sessions — sites, backup architecture, governing principles,
gotchas, where-to-look guide. Pointed at from CLAUDE.md.
This commit is contained in:
@@ -0,0 +1,99 @@
|
||||
# pbs-nh3
|
||||
|
||||
Proxmox Backup Server — **DR mirror**. VM on nh3-pve with an
|
||||
NFS-mounted datastore on the NH3 Synology. Pulls nightly from pbs-ana
|
||||
(the primary) at 06:00.
|
||||
|
||||
## Network
|
||||
|
||||
- **LAN IP:** 10.100.50.90
|
||||
- **FQDN:** `pbs-nh3.phasefinal.com`
|
||||
- **PBS web UI:** <https://10.100.50.90:8007/>
|
||||
- **SSH:** `ssh pbs-nh3` (config alias → `lkraven@10.100.50.90`).
|
||||
Root ops via sudo from lkraven.
|
||||
- **Homepage card:** *PBS-NH3* in `Infra - NH3` group.
|
||||
|
||||
## Hardware (VM)
|
||||
|
||||
- **Hypervisor:** nh3-pve
|
||||
- **vCPU:** 4 (QEMU)
|
||||
- **RAM:** 7.8 GB
|
||||
- **Root disk:** `/dev/sda1` 31 GB ext4 — OS only (2 GB used)
|
||||
- **Datastore:** **NFS from nh3-nas** at `/mnt/pbs-datastore` →
|
||||
`10.100.50.50:/volume1/pbs`. Volume size 42 TB; 27 TB used (64%).
|
||||
|
||||
## NFS details (the Synology ACL gotcha)
|
||||
|
||||
Synology's `/volume1/pbs` share was initially created with "Advanced
|
||||
Permissions" which imposes a `syno_acl` that denies writes to
|
||||
non-admin UIDs (even the `backup` uid PBS runs as). Fix was to flatten
|
||||
the share to pure POSIX mode 777 so the mount becomes "Linux mode"
|
||||
(`synoacltool -get` confirms). `no_root_squash + no_all_squash` also
|
||||
set on the NFS export.
|
||||
|
||||
Mount uses NFSv3 (same pattern as pbs-ana, for consistency more than
|
||||
necessity — Synology's Btrfs doesn't have the ZFS-case-insensitivity
|
||||
issue):
|
||||
|
||||
```
|
||||
10.100.50.50:/volume1/pbs /mnt/pbs-datastore nfs \
|
||||
defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=3 0 0
|
||||
```
|
||||
|
||||
Full history: `docs/runbooks/pbs-deployment.md` Phase 5.3.
|
||||
|
||||
## What runs here
|
||||
|
||||
- `proxmox-backup-proxy` — HTTPS API + web UI (port 8007)
|
||||
- `proxmox-backup` — backend service
|
||||
- Scheduled verify job: weekly Sunday 12:00 (after nightly sync
|
||||
completes), `ignore-verified=true`, `outdated-after=30 days`
|
||||
- Scheduled sync from pbs-ana: daily at 06:00 UTC
|
||||
|
||||
## Remote config
|
||||
|
||||
- Remote name: `pbs-ana`
|
||||
- Remote host: `10.250.50.90:8007`
|
||||
- Auth: `root@pam!fleet-vzdump` token (same token as clients use on
|
||||
the primary — shared here for simplicity; could be split to a
|
||||
sync-only read-only token)
|
||||
- Cert fingerprint: in password manager under "pbs-ana fingerprint"
|
||||
|
||||
## Namespaces
|
||||
|
||||
Namespaces are auto-replicated from pbs-ana via the sync job — you do
|
||||
NOT need to pre-create them on the mirror. Expected namespaces after
|
||||
first sync: same 5 as pbs-ana (`pfi-pve`, `nh3-pve`, `esh-pve`,
|
||||
`esh-pve-nas`, `sfsrv-ana`).
|
||||
|
||||
## Dependencies / blast radius
|
||||
|
||||
- **Hard dependency on nh3-nas** (Synology). Datastore goes dead if
|
||||
the Synology crashes; sync fails, verify fails, restore ops fail.
|
||||
But: ANA primary keeps operating fine, so the fleet's backup SLA
|
||||
is preserved — only DR redundancy is temporarily lost.
|
||||
- **Sync source**: pbs-ana going offline means no new snapshots pull
|
||||
here, but existing ones remain restorable. Sync resumes
|
||||
automatically once pbs-ana recovers.
|
||||
|
||||
## DR failover (hypothetical)
|
||||
|
||||
If pbs-ana is permanently lost and we needed to restore fleet VMs
|
||||
from pbs-nh3:
|
||||
|
||||
1. Re-point each PVE host's `pbs-ana` storage entry at pbs-nh3
|
||||
(`pve-datacenter.cfg` → change `server` to 10.100.50.90)
|
||||
2. Fingerprint + token still work (same token)
|
||||
3. Cross-WAN restore is slower than local but functional
|
||||
|
||||
Documented as future work — not tested. Run a trial restore of a
|
||||
small CT periodically to validate.
|
||||
|
||||
## Refresh
|
||||
|
||||
```
|
||||
scripts/refresh-server-info.sh pbs-nh3
|
||||
```
|
||||
|
||||
Snapshot at `system-details.txt`. Full deployment history:
|
||||
`docs/runbooks/pbs-deployment.md`.
|
||||
@@ -0,0 +1 @@
|
||||
lkraven@10.100.50.90
|
||||
@@ -0,0 +1,84 @@
|
||||
|
||||
===== HOST =====
|
||||
|
||||
Hostname: pbs-nh3.phasefinal.com
|
||||
Date: 2026-04-23T21:56:13-07:00
|
||||
Uptime: up 22 hours, 41 minutes
|
||||
OS: Debian GNU/Linux 12 (bookworm)
|
||||
Kernel: 6.1.0-44-amd64
|
||||
Arch: x86_64
|
||||
|
||||
===== HARDWARE =====
|
||||
|
||||
CPU cores: 4
|
||||
CPU model: QEMU Virtual CPU version 2.5+
|
||||
MemTotal: 7.8 GB
|
||||
MemAvailable: 7.3 GB
|
||||
|
||||
===== GPUS =====
|
||||
|
||||
nvidia-smi not present (no NVIDIA GPUs or driver not installed)
|
||||
|
||||
===== FILESYSTEMS (df) =====
|
||||
|
||||
Filesystem Size Used Avail Use% Mounted on
|
||||
/dev/sda1 31G 2.0G 27G 7% /
|
||||
10.100.50.50:/volume1/pbs 42T 27T 16T 64% /mnt/pbs-datastore
|
||||
|
||||
===== PERSISTENT MOUNTS (/etc/fstab, non-comment) =====
|
||||
|
||||
UUID=34a61420-a414-4789-8a3d-065a8905eb11 / ext4 errors=remount-ro 0 1
|
||||
UUID=518afc03-e17d-4e1a-a5d4-84b96bee1c60 none swap sw 0 0
|
||||
/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0
|
||||
10.100.50.50:/volume1/pbs /mnt/pbs-datastore nfs defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=3 0 0
|
||||
|
||||
===== TARGETED DATA PATHS =====
|
||||
|
||||
/opt (total: 4.0K)
|
||||
total 8
|
||||
drwxr-xr-x 2 root root 4096 2026-04-22 21:04 .
|
||||
drwxr-xr-x 18 root root 4096 2026-04-22 21:06 ..
|
||||
|
||||
/srv (total: 4.0K)
|
||||
total 8
|
||||
drwxr-xr-x 2 root root 4096 2026-04-22 21:04 .
|
||||
drwxr-xr-x 18 root root 4096 2026-04-22 21:06 ..
|
||||
|
||||
|
||||
===== DOCKER =====
|
||||
|
||||
docker not installed
|
||||
|
||||
===== COMPOSE FILES (/opt/docker/compose/) =====
|
||||
|
||||
/opt/docker/compose not present
|
||||
|
||||
===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) =====
|
||||
|
||||
/opt/docker/conf not present
|
||||
|
||||
===== LISTENING PORTS =====
|
||||
|
||||
0.0.0.0:111
|
||||
0.0.0.0:22
|
||||
0.0.0.0:41669
|
||||
0.0.0.0:46653
|
||||
[::]:111
|
||||
[::1]:25
|
||||
127.0.0.1:25
|
||||
127.0.0.1:82
|
||||
[::]:22
|
||||
[::]:37707
|
||||
[::]:54621
|
||||
*:8007
|
||||
|
||||
===== MODEL / HUGGINGFACE CACHES =====
|
||||
|
||||
|
||||
===== DOCKER-ADJACENT SYSTEMD SERVICES =====
|
||||
|
||||
(none matching)
|
||||
|
||||
===== DONE =====
|
||||
|
||||
Paste the above back into the chat, or pass a path as argv[1] to save.
|
||||
Reference in New Issue
Block a user