From 0368ab732ad23bbec8746055952b2606bc50b280 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 21 Apr 2026 17:14:46 -0700 Subject: [PATCH] docs: PBS deployment runbook (ANA primary + NH3 DR mirror) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit End-to-end runbook for standing up Proxmox Backup Server across the fleet. Path A architecture: single primary at ANA, one-way sync to NH3 for disaster recovery. All 5 hypervisors (pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) migrate from local-dump vzdump to PBS-ANA. Key decisions captured in the runbook: - PBS in a Debian VM (not LXC) for clean capability model. - PBS-ANA on pfi-pve, datastore via NFS from 10.250.50.50 — separates backup data from hypervisor boot disk. - PBS-NH3 on nh3-pve with local storage (independent failure domain from ANA). - Dedicated fleet-vzdump API token; read-only sync token for PBS-NH3's pull job. - sfsrv-ana specifically goes from zero backup coverage to full vzdump coverage in Phase 3. 9 phases, each self-contained with a done-state and rollback posture. User can stop between phases without leaving the fleet in a bad state. STATUS.md: added item 6b tracking this deployment. Original item 6 (cross-site rsync) now scoped to restic-only since PBS handles the VM-image cross-site redundancy directly. --- STATUS.md | 15 + docs/runbooks/pbs-deployment.md | 476 ++++++++++++++++++++++++++++++++ 2 files changed, 491 insertions(+) create mode 100644 docs/runbooks/pbs-deployment.md diff --git a/STATUS.md b/STATUS.md index f4f8c93..fe7414c 100644 --- a/STATUS.md +++ b/STATUS.md @@ -91,6 +91,21 @@ significant work lands — don't let it drift quietly. (`/mnt/backup/restic/repo/ana/`) and NH3 Synology data dir. Planned since initial rest-server setup; not built. Needs Synology SSH access first (item 8). ~20 min once access is there. + - Note: cross-site mirroring for the VM-image layer is being + addressed by the PBS deployment (item 6b) — this rsync is now + scoped to restic repos only. + +6b. **PBS deployment across the fleet.** Runbook lives at + `docs/runbooks/pbs-deployment.md`. Primary at ANA (VM on pfi-pve, + NFS datastore on Debian NAS), DR mirror at NH3 (VM on nh3-pve, + local datastore). One-way sync from ANA → NH3 nightly. All 5 + hypervisors (pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) + migrate off local-dump vzdump jobs onto PBS. + - Closes: SureFire's zero-coverage gap (currently no vzdump at all on sfsrv-ana) + - Closes: "cross-site redundancy for VM images" scope item + - Provides: cross-fleet block-level dedupe, dirty-bitmap incrementals, + real safe-prune retention, per-host encryption keys + - Est. 4–6h spread across sittings. 9 phases in the runbook. 7. **SureFire tenant backup plan decision.** Three options documented in `servers/sfsrv-ana/README.md`: diff --git a/docs/runbooks/pbs-deployment.md b/docs/runbooks/pbs-deployment.md new file mode 100644 index 0000000..3047486 --- /dev/null +++ b/docs/runbooks/pbs-deployment.md @@ -0,0 +1,476 @@ +# Proxmox Backup Server deployment runbook + +Stand up PBS-ANA as the fleet's primary vzdump target, then PBS-NH3 as +a DR mirror that pulls nightly from PBS-ANA. Migrate all 5 hypervisors +(pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) off their current +local-dump vzdump targets onto PBS-ANA. + +Architecture decision + rationale: see `STATUS.md` and the earlier +session transcript. Short version: one primary at ANA, one-way sync to +NH3, ANA owns retention policy. + +**Estimated total time: 4–6 hours spread across sittings.** Each phase +below is a self-contained work unit with a clear done-state; you can +stop after any phase without leaving the fleet in a bad state. + +## Phase 0 — Prerequisites (one-time, before Phase 1) + +Check once: + +```bash +# Debian NAS (10.250.50.50) is exporting to pfi-pve already for +# rest-server-ana's /mnt/backup. We need to add a sibling export for +# PBS-ANA's datastore. SSH to the NAS: +ssh ana-nas 'cat /etc/exports' +``` + +Make sure `/mnt/backup` export allows pfi-pve (10.250.250.31) to mount +read-write. If it does (which it should, per the existing +rest-server-ana setup), we're good. + +Decide up front: + +- **PBS-ANA datastore path:** recommend `/mnt/backup/pbs-ana/` (sibling + to `/mnt/backup/restic/`). Create the dir on the NAS with the user + that PBS will run as (root in the VM maps to root on NFS if + `no_root_squash` is set — matches the rest-server-ana pattern). +- **PBS-NH3 datastore path:** local-only on nh3-pve. Recommend a new + Proxmox dir storage on a data disk; `/mnt/pbs-nh3/datastore/`. + +## Phase 1 — Provision PBS-ANA (VM on pfi-pve) + +**Target state:** a fresh Debian 12 VM on pfi-pve running +`proxmox-backup-server`, with a datastore configured on the NFS-mounted +NAS path. Reachable at `pbs-ana.phasefinal.com` (or by IP). + +### 1.1. Create a Debian 12 VM on pfi-pve + +Via the pve web UI or CLI. Starting specs: + +| Setting | Value | +|---|---| +| Name | `pbs-ana` | +| VMID | next available (probably 114 given pfi-pve's fleet) | +| ISO | debian-12.x.x-amd64-netinst | +| CPU | 4 cores | +| RAM | 8 GB (can be tight at scale; bump if needed) | +| Disk | 32 GB on `local-lvm` (for OS only — datastore is NFS) | +| Network | vmbr0, firewall on | +| Onboot | yes | + +Install Debian 12 minimal (no desktop, SSH server, standard system utilities). + +Once up, set a static IP in the `servers/` LAN range (e.g. +`10.250.50.90`) or DHCP-reserved for stability. + +### 1.2. Install PBS package + +SSH in as root: + +```bash +# Add the PBS no-subscription repo +cat > /etc/apt/sources.list.d/pbs-no-subscription.list <<'EOF' +deb http://download.proxmox.com/debian/pbs bookworm pbs-no-subscription +EOF + +wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg \ + -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg + +apt update +apt full-upgrade -y +apt install -y proxmox-backup-server +``` + +Reboot once the PBS kernel is installed (PBS kernel is Proxmox's +patched one, like PVE hosts): + +```bash +systemctl reboot +``` + +After reboot, verify the web UI is reachable: + +``` +https://10.250.50.90:8007/ +``` + +Default login: `root` / (the Debian root password you set at install). + +### 1.3. Mount NAS datastore path + +On the PBS VM: + +```bash +# Create mount point +mkdir -p /mnt/pbs-datastore + +# Add persistent NFS mount +cat >> /etc/fstab <<'EOF' +10.250.50.50:/mnt/backup/pbs-ana /mnt/pbs-datastore nfs4 defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=4.2 0 0 +EOF + +# Create the dir on NAS side if it doesn't exist +ssh ana-nas 'mkdir -p /mnt/backup/pbs-ana && chown root:root /mnt/backup/pbs-ana' + +# Mount +mount -a +mount | grep pbs-datastore +# Should show the NFS mount active +df -h /mnt/pbs-datastore +``` + +### 1.4. Create PBS datastore + +In the PBS web UI: **Datastore → Add Datastore**: + +| Field | Value | +|---|---| +| Name | `backups` | +| Backing Path | `/mnt/pbs-datastore` | +| Retention (can set later) | — | +| GC Schedule | Sunday 06:00 | +| Prune Schedule | Daily 05:30 | + +The first time PBS writes to the datastore it initializes `.chunks/`, +`.lock`, etc. Takes a few seconds. + +### 1.5. Create an API token for hypervisors to use + +PBS web UI: **Configuration → Access Control → API Token → Add**: + +| Field | Value | +|---|---| +| User | `root@pam` | +| Token Name | `fleet-vzdump` | +| Privilege Separation | **off** (so the token inherits root perms) | + +Save the token secret — this is the only time PBS shows it. Store in +your password manager alongside the other fleet creds. + +Token identifier looks like: `root@pam!fleet-vzdump` +Secret looks like: `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` + +### 1.6. Note PBS fingerprint + +For client connections: + +```bash +# On PBS VM +proxmox-backup-manager cert info | grep -i fingerprint +``` + +Save this fingerprint — each PVE client needs it to verify PBS's cert. + +### Phase 1 done-state + +- PBS-ANA reachable at `https://:8007/` +- Datastore `backups` created and healthy +- API token `root@pam!fleet-vzdump` issued +- Cert fingerprint recorded + +## Phase 2 — Migrate pfi-pve's vzdump jobs onto PBS-ANA + +**Target state:** pfi-pve sends its nightly backups to PBS-ANA. + +### 2.1. Add PBS-ANA as storage on pfi-pve + +Web UI: **Datacenter → Storage → Add → Proxmox Backup Server**: + +| Field | Value | +|---|---| +| ID | `pbs-ana` | +| Server | PBS-ANA IP | +| Username | `root@pam!fleet-vzdump` | +| Password | the token secret from 1.5 | +| Datastore | `backups` | +| Fingerprint | from 1.6 | +| Content | VZDump backup file | +| Nodes | `pve` (pfi-pve) | + +Click Add. Storage should show as available within seconds. + +### 2.2. Test backup of one VM + +Pick a small VM (e.g. an idle LXC like ana-filebot, VMID 112). Web UI: +**select VM → Backup → Backup Now → Storage: pbs-ana**. + +First backup is a full dump. For a small LXC, a few minutes. Verify: +- Backup completes with green OK status +- In PBS UI: **Datastore → backups** shows the snapshot under + `ct/112/` or `vm/112/` + +### 2.3. Migrate the scheduled backup job + +Web UI: **Datacenter → Backup**. Edit the existing pfi-pve backup job +(`backup-4b911ca7-92df` from the audit). + +| Change | From | To | +|---|---|---| +| Storage | `pve-truenas` (or current) | `pbs-ana` | +| Mode | snapshot | snapshot | +| Compression | zstd | zstd | + +Save. Next scheduled run (03:00 by default) fires against PBS-ANA. + +### 2.4. Run it manually to verify + +Web UI: **backup job → Run Now**. Watch each VM's backup complete. +Larger VMs (PFI-Postgres 8GB RAM, PFI-Mongo, etc.) take 10-30 minutes +each on first full backup. Subsequent backups will be minutes due to +dirty-bitmap incrementals. + +### 2.5. Keep legacy target for overlap + +**Don't delete the old `pve-truenas` backup storage yet.** Wait ~1 week +of successful PBS runs before retiring it. Double-coverage period is +cheap insurance. + +### Phase 2 done-state + +- pfi-pve's 11 guests have first PBS snapshots +- Scheduled job uses PBS-ANA +- Old `pve-truenas` target still exists (for safety) but no new jobs + write to it + +## Phase 3 — Onboard esh-pve, esh-pve-nas, sfsrv-ana + +**Target state:** all three remaining ANA/ESH hypervisors use PBS-ANA. + +### 3.1. Repeat Phase 2.1 on each hypervisor + +On each of `esh-pve`, `esh-pve-nas`, `sfsrv-ana`: + +- Web UI: Datacenter → Storage → Add → Proxmox Backup Server (same + fields as Phase 2.1; same token is fine — it's fleet-wide). + +### 3.2. Configure vzdump jobs + +`esh-pve` and `esh-pve-nas` already have vzdump jobs hitting local +dump dirs. Edit them per Phase 2.3. + +`sfsrv-ana` has **no existing vzdump jobs** (per audit). Create one: + +Web UI: Datacenter → Backup → Add: + +| Field | Value | +|---|---| +| Storage | `pbs-ana` | +| Schedule | `03:00` (daily) | +| Selection | All | +| Retention | keep-daily 7, weekly 4, monthly 12, yearly 3 | +| Compression | zstd | +| Mode | snapshot | +| Email | your address, notifications on failure | + +### 3.3. First backups + +Fire manual runs from each hypervisor. ESH site backups cross-WAN to +ANA — first full dumps will take a while. Schedule overnight if +bandwidth is an issue during the day. + +### 3.4. Verify on PBS-ANA side + +Web UI on PBS: **Datastore → backups → Content**. Should show +namespaces / groups for guests from all 4 hypervisors by hostname-ish +prefixes (PBS uses VMID + node). + +### Phase 3 done-state + +- All 3 hypervisors have working PBS backups +- sfsrv-ana specifically goes from ZERO backup coverage to full + vzdump coverage (biggest single gain of this migration) + +## Phase 4 — Onboard nh3-pve + +**Target state:** nh3-pve uses PBS-ANA as backup target despite the +WAN hop. + +Same process as Phase 3. The only extra considerations: + +- **Bandwidth:** first full backups of NH3 VMs cross the WAN to ANA. + Estimate size and run during off-hours. Incrementals after that + should be small. +- **Verify WAN stability:** if the link flaps during a backup, PBS + retries automatically but a multi-hour drop will fail. Don't start + a 50 GB backup during known-unstable windows. + +### Phase 4 done-state + +- All 5 hypervisors back up to PBS-ANA +- Verified: ~24 guests total across the fleet, all with recent + PBS snapshots + +## Phase 5 — Provision PBS-NH3 (DR mirror) + +**Target state:** PBS-NH3 running on nh3-pve, ready to pull from +PBS-ANA. + +### 5.1. Create Debian VM on nh3-pve + +Same pattern as Phase 1.1: + +| Setting | Value | +|---|---| +| Name | `pbs-nh3` | +| VMID | next available on nh3-pve | +| CPU | 2 cores | +| RAM | 4 GB | +| Disk | 32 GB OS + data disk (see 5.3) | +| Network | vmbr0 | + +### 5.2. Install PBS + +Same as Phase 1.2. + +### 5.3. Local datastore + +PBS-NH3 uses local storage rather than NFS (keeps the two PBSes on +different failure domains). Two options: + +**Option A:** attach a second virtual disk to the VM (e.g. 2 TB on +nh3-pve's `local-lvm` or whatever storage is available), format ext4, +mount at `/mnt/pbs-datastore`. + +**Option B:** mount Synology Btrfs share via NFS or CIFS. Simpler +storage admin but couples PBS-NH3 to the Synology's availability. + +Recommended: **Option A**. Keeps it self-contained. + +```bash +# Inside the PBS-NH3 VM, after attaching a disk /dev/sdb: +mkfs.ext4 /dev/sdb +mkdir -p /mnt/pbs-datastore +echo '/dev/sdb /mnt/pbs-datastore ext4 defaults 0 2' >> /etc/fstab +mount -a +``` + +### 5.4. Create datastore + +Web UI: Datastore → Add, name `backups-mirror`, path `/mnt/pbs-datastore`. + +### Phase 5 done-state + +- PBS-NH3 reachable, datastore ready +- No sync job yet — that's Phase 6 + +## Phase 6 — Sync PBS-ANA → PBS-NH3 + +**Target state:** PBS-NH3 nightly pulls everything from PBS-ANA. + +### 6.1. Create a remote entry on PBS-NH3 + +PBS-NH3 web UI: **Configuration → Remotes → Add**: + +| Field | Value | +|---|---| +| Remote ID | `pbs-ana` | +| Host | PBS-ANA IP | +| User ID | `root@pam!fleet-vzdump` (or a dedicated read-only token — see 6.3) | +| Password | token secret | +| Fingerprint | PBS-ANA's cert fingerprint | + +### 6.2. Create a sync job + +Web UI: **Datastore → backups-mirror → Sync Jobs → Add**: + +| Field | Value | +|---|---| +| Remote | `pbs-ana` | +| Remote Datastore | `backups` | +| Sync Direction | Pull | +| Schedule | `*-*-* 06:00:00` (after primary backups at 03:00) | +| Max Depth | unlimited | +| Owner | `root@pam` | + +Save. Run once manually to confirm. + +First run transfers everything the primary has. This is one-time heavy +— schedule on a weekend if bandwidth is a concern. + +### 6.3. (Optional) tighten the token + +The `fleet-vzdump` token has write permissions on PBS-ANA. For +sync-only access, create a second read-only token on PBS-ANA +(`Datastore.Audit` + `Remote.Audit`) and use that on the PBS-NH3 +remote config. Reduces blast radius if PBS-NH3 ever compromised. + +### 6.4. Set retention on PBS-NH3 + +Shorter than primary (it's DR, not archival): + +Web UI: Datastore → backups-mirror → Prune & GC → Edit: + +| Field | Value | +|---|---| +| keep-last | 7 | +| keep-daily | 7 | +| keep-weekly | 4 | + +### Phase 6 done-state + +- Nightly pull from PBS-ANA → PBS-NH3 runs cleanly +- PBS-NH3 holds last 7 days of backups (shorter retention, DR focus) + +## Phase 7 — Burn-in + +**Target state:** ~1 week of clean operation before retiring legacy targets. + +Daily checks: + +```bash +# PBS-ANA datastore growth is sensible +ssh root@pbs-ana 'df -h /mnt/pbs-datastore' + +# PBS-NH3 sync job succeeded overnight +ssh root@pbs-nh3 'journalctl -u proxmox-backup-proxy --since yesterday | grep -i sync' + +# Every hypervisor's last backup is ≤24h old +# (Run refresh-proxmox-info.sh and inspect the backup-coverage section) +scripts/refresh-proxmox-info.sh all +``` + +Watch for: + +- Failed backup jobs (email alerts if enabled) +- Sync job duration growing unexpectedly (bandwidth issue) +- Datastore approaching capacity (bump retention or storage) +- VMs that don't dirty-bitmap incremental (missing QEMU Guest Agent, + shut-down VMs always do full dumps) + +## Phase 8 — Retire legacy vzdump targets + +**Target state:** no more old-style local-dump backups running; just PBS. + +Only after Phase 7 passes: + +- On each hypervisor: remove old vzdump jobs that targeted local dump + dirs. +- Optionally delete old VMA files from local dump storage to reclaim + space (before deleting, verify PBS has the same vintage). +- Update `servers/*-pve/proxmox-details.txt` by re-running + `scripts/refresh-proxmox-info.sh all` — the `backup coverage` section + should now show every guest covered by the PBS-ANA job. + +## Phase 9 — Update this workspace + +- Update `STATUS.md` coverage table to show PBS deployment state. +- Update memory `project_backup_pipeline_gaps.md` — PBS closes several + of the open gaps. +- Update `README.md` backup pipeline section to describe the new + topology. +- Add an entry for PBS-ANA and PBS-NH3 in `servers/` (with README + + ssh-target). They're first-class infrastructure now. +- Homepage Infra-ANA card pointing at PBS-ANA Web UI; Infra-NH3 for + PBS-NH3. + +## Rollback posture + +At any phase, you can stop and leave the fleet in its current state: + +- Phase 1 incomplete: no impact, PBS-ANA just sits idle. +- Phase 2+ incomplete: affected hypervisors still have their old + vzdump jobs running. Until Phase 8, old targets are kept as + redundant. Rolling back one hypervisor means deleting its + PBS storage entry and re-enabling its old job. +- Phase 6+ incomplete: PBS-ANA still provides primary backup coverage. + DR mirror just isn't built yet.