pbs-nh3

2026-10-04: PBS 4.2.7 on Debian 13.7 (kernel 6.12.111), upgraded from 3.4.8 (docs/runbooks/pbs-pbs4-upgrade-plan.md). PBS-on-plain-Debian install (no proxmox-backup meta package, Debian kernel), so pbs3to4 FAILs one check by design. ssh infra-ops@10.100.50.90 (uid 850, fleet key, NOPASSWD sudo with log_output) since 2026-10-04 (Prime); the hypervisor's guest agent (qm guest exec) is the fallback if the network is down.

Proxmox Backup Server — DR mirror. VM on nh3-pve with an NFS-mounted datastore on the NH3 Synology. Pulls nightly from pbs-ana (the primary) at 06:00.

Network

LAN IP: 10.100.50.90
FQDN: pbs-nh3.phasefinal.com
PBS web UI: https://10.100.50.90:8007/
SSH: ssh pbs-nh3 (config alias → lkraven@10.100.50.90). Root ops via sudo from lkraven.
Homepage card: PBS-NH3 in Infra - NH3 group.

Hardware (VM)

Hypervisor: nh3-pve
vCPU: 4 (QEMU)
RAM: 7.8 GB
Root disk: /dev/sda1 31 GB ext4 — OS only (2 GB used)
Datastore: NFS from nh3-nas at /mnt/pbs-datastore → 10.100.50.50:/volume1/pbs. Volume size 42 TB; 27 TB used (64%).

NFS details (the Synology ACL gotcha)

Synology's /volume1/pbs share was initially created with "Advanced Permissions" which imposes a syno_acl that denies writes to non-admin UIDs (even the backup uid PBS runs as). Fix was to flatten the share to pure POSIX mode 777 so the mount becomes "Linux mode" (synoacltool -get confirms). no_root_squash + no_all_squash also set on the NFS export.

Mount uses NFSv3 (same pattern as pbs-ana, for consistency more than necessity — Synology's Btrfs doesn't have the ZFS-case-insensitivity issue):

10.100.50.50:/volume1/pbs /mnt/pbs-datastore nfs \
  defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=3 0 0

Full history: docs/runbooks/pbs-deployment.md Phase 5.3.

What runs here

proxmox-backup-proxy — HTTPS API + web UI (port 8007)
proxmox-backup — backend service
Scheduled verify job: weekly Sunday 12:00 (after nightly sync completes), ignore-verified=true, outdated-after=30 days
Scheduled sync from pbs-ana: daily at 06:00 UTC

Remote config

Remote name: pbs-ana
Remote host: 10.250.50.90:8007
Auth: root@pam!fleet-vzdump token (same token as clients use on the primary — shared here for simplicity; could be split to a sync-only read-only token)
Cert fingerprint: in password manager under "pbs-ana fingerprint"

Namespaces

Namespaces are auto-replicated from pbs-ana via the sync job — you do NOT need to pre-create them on the mirror. Expected namespaces after first sync: same 5 as pbs-ana (pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana).

Dependencies / blast radius

Hard dependency on nh3-nas (Synology). Datastore goes dead if the Synology crashes; sync fails, verify fails, restore ops fail. But: ANA primary keeps operating fine, so the fleet's backup SLA is preserved — only DR redundancy is temporarily lost.
Sync source: pbs-ana going offline means no new snapshots pull here, but existing ones remain restorable. Sync resumes automatically once pbs-ana recovers.

DR failover (hypothetical)

If pbs-ana is permanently lost and we needed to restore fleet VMs from pbs-nh3:

Re-point each PVE host's pbs-ana storage entry at pbs-nh3 (pve-datacenter.cfg → change server to 10.100.50.90)
Fingerprint + token still work (same token)
Cross-WAN restore is slower than local but functional

Documented as future work — not tested. Run a trial restore of a small CT periodically to validate.

Refresh

scripts/refresh-server-info.sh pbs-nh3

Snapshot at system-details.txt. Full deployment history: docs/runbooks/pbs-deployment.md.

3.7 KiB Raw Blame History