esh-pfi-infrastructure/docs/runbooks/pbs-deployment.md

Proxmox Backup Server deployment runbook

Stand up PBS-ANA as the fleet's primary vzdump target, then PBS-NH3 as a DR mirror that pulls nightly from PBS-ANA. Migrate all 5 hypervisors (pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) off their current local-dump vzdump targets onto PBS-ANA.

Architecture decision + rationale: see STATUS.md and the earlier session transcript. Short version: one primary at ANA, one-way sync to NH3, ANA owns retention policy.

Estimated total time: 4–6 hours spread across sittings. Each phase below is a self-contained work unit with a clear done-state; you can stop after any phase without leaving the fleet in a bad state.

Phase 0 — Prerequisites (one-time, before Phase 1)

0.1. Check current NFS exports

ssh -t 10.250.50.50 'sudo exportfs -v'

The existing broad exports (e.g. /mnt/backup → 10.0.0.0/8) cover pfi-pve but use root_squash — which will block PBS from writing its datastore metadata. We need a dedicated, no-squash export scoped to the PBS-ANA VM's IP.

0.2. Create a dedicated export for PBS-ANA

Assign PBS-ANA its LAN IP (let's say 10.250.50.90 — Phase 1.1 uses this), then on the Debian NAS:

ssh -t 10.250.50.50 '
  sudo mkdir -p /mnt/backup/pbs-ana &&
  sudo chown root:root /mnt/backup/pbs-ana &&
  sudo chmod 755 /mnt/backup/pbs-ana &&

  echo "/mnt/backup/pbs-ana  10.250.50.90(rw,sync,no_root_squash,no_subtree_check,sec=sys)" \
    | sudo tee /etc/exports.d/pbs-ana.exports > /dev/null &&

  sudo exportfs -ra &&
  sudo exportfs -v | grep pbs-ana
'

Should print an /mnt/backup/pbs-ana line with no_root_squash.

The existing cockpit-file-sharing.exports is managed by Cockpit's File Sharing module; keep the PBS-specific export in a separate file so Cockpit doesn't clobber it.

0.3. Decide datastore paths

• PBS-ANA datastore path: /mnt/backup/pbs-ana/ on the NAS, mounted at /mnt/pbs-datastore/ inside the PBS-ANA VM (Phase 1.3).
• PBS-NH3 datastore path: local-only on nh3-pve, mounted at /mnt/pbs-datastore/ inside the PBS-NH3 VM (Phase 5.3).

0.4. Side-note: /mnt/pve-VMStorage is exported to <world>

Not blocking PBS work but worth tightening during a Cockpit pass — reduce to 10.0.0.0/8 or more specific. Open item, not blocker.

Phase 1 — Provision PBS-ANA (VM on pfi-pve)

Target state: a fresh Debian 12 VM on pfi-pve running proxmox-backup-server, with a datastore configured on the NFS-mounted NAS path. Reachable at pbs-ana.phasefinal.com (or by IP).

1.1. Create a Debian 12 VM on pfi-pve

Via the pve web UI or CLI. Starting specs:

Setting	Value
Name	pbs-ana
VMID	next available (probably 114 given pfi-pve's fleet)
ISO	debian-12.x.x-amd64-netinst
CPU	4 cores
RAM	8 GB (can be tight at scale; bump if needed)
Disk	32 GB on local-lvm (for OS only — datastore is NFS)
Network	vmbr0, firewall on
Onboot	yes

Install Debian 12 minimal (no desktop, SSH server, standard system utilities).

Once up, set a static IP in the servers/ LAN range (e.g. 10.250.50.90) or DHCP-reserved for stability.

1.2. Install PBS package

SSH in as root:

# Add the PBS no-subscription repo
cat > /etc/apt/sources.list.d/pbs-no-subscription.list <<'EOF'
deb http://download.proxmox.com/debian/pbs bookworm pbs-no-subscription
EOF

wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg \
  -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg

apt update
apt full-upgrade -y
apt install -y proxmox-backup-server

Reboot once the PBS kernel is installed (PBS kernel is Proxmox's patched one, like PVE hosts):

systemctl reboot

After reboot, verify the web UI is reachable:

https://10.250.50.90:8007/

Default login: root / (the Debian root password you set at install).

1.3. Mount NAS datastore path

On the PBS VM:

# Create mount point
mkdir -p /mnt/pbs-datastore

# Add persistent NFS mount
cat >> /etc/fstab <<'EOF'
10.250.50.50:/mnt/backup/pbs-ana  /mnt/pbs-datastore  nfs4  defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=4.2  0  0
EOF

# Create the dir on NAS side if it doesn't exist
ssh ana-nas 'mkdir -p /mnt/backup/pbs-ana && chown root:root /mnt/backup/pbs-ana'

# Mount
mount -a
mount | grep pbs-datastore
# Should show the NFS mount active
df -h /mnt/pbs-datastore

1.4. Create PBS datastore

In the PBS web UI: Datastore → Add Datastore:

Field	Value
Name	backups
Backing Path	/mnt/pbs-datastore
Retention (can set later)	—
GC Schedule	Sunday 06:00
Prune Schedule	Daily 05:30

The first time PBS writes to the datastore it initializes .chunks/, .lock, etc. Takes a few seconds.

1.5. Create an API token for hypervisors to use

PBS web UI: Configuration → Access Control → API Token → Add:

Field	Value
User	root@pam
Token Name	fleet-vzdump
Privilege Separation	off (so the token inherits root perms)

Save the token secret — this is the only time PBS shows it. Store in your password manager alongside the other fleet creds.

Token identifier looks like: root@pam!fleet-vzdump Secret looks like: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

1.6. Note PBS fingerprint

For client connections:

# On PBS VM
proxmox-backup-manager cert info | grep -i fingerprint

Save this fingerprint — each PVE client needs it to verify PBS's cert.

Phase 1 done-state

• PBS-ANA reachable at https://<ip>:8007/
• Datastore backups created and healthy
• API token root@pam!fleet-vzdump issued
• Cert fingerprint recorded

Phase 2 — Migrate pfi-pve's vzdump jobs onto PBS-ANA

Target state: pfi-pve sends its nightly backups to PBS-ANA.

2.1. Add PBS-ANA as storage on pfi-pve

Web UI: Datacenter → Storage → Add → Proxmox Backup Server:

Field	Value
ID	pbs-ana
Server	PBS-ANA IP
Username	root@pam!fleet-vzdump
Password	the token secret from 1.5
Datastore	backups
Fingerprint	from 1.6
Content	VZDump backup file
Nodes	pve (pfi-pve)

Click Add. Storage should show as available within seconds.

2.2. Test backup of one VM

Pick a small VM (e.g. an idle LXC like ana-filebot, VMID 112). Web UI: select VM → Backup → Backup Now → Storage: pbs-ana.

First backup is a full dump. For a small LXC, a few minutes. Verify:

• Backup completes with green OK status
• In PBS UI: Datastore → backups shows the snapshot under ct/112/ or vm/112/

2.3. Migrate the scheduled backup job

Web UI: Datacenter → Backup. Edit the existing pfi-pve backup job (backup-4b911ca7-92df from the audit).

Change	From	To
Storage	pve-truenas (or current)	pbs-ana
Mode	snapshot	snapshot
Compression	zstd	zstd

Save. Next scheduled run (03:00 by default) fires against PBS-ANA.

2.4. Run it manually to verify

Web UI: backup job → Run Now. Watch each VM's backup complete. Larger VMs (PFI-Postgres 8GB RAM, PFI-Mongo, etc.) take 10-30 minutes each on first full backup. Subsequent backups will be minutes due to dirty-bitmap incrementals.

2.5. Keep legacy target for overlap

Don't delete the old pve-truenas backup storage yet. Wait ~1 week of successful PBS runs before retiring it. Double-coverage period is cheap insurance.

Phase 2 done-state

• pfi-pve's 11 guests have first PBS snapshots
• Scheduled job uses PBS-ANA
• Old pve-truenas target still exists (for safety) but no new jobs write to it

Phase 3 — Onboard esh-pve, esh-pve-nas, sfsrv-ana

Target state: all three remaining ANA/ESH hypervisors use PBS-ANA.

3.1. Repeat Phase 2.1 on each hypervisor

On each of esh-pve, esh-pve-nas, sfsrv-ana:

• Web UI: Datacenter → Storage → Add → Proxmox Backup Server (same fields as Phase 2.1; same token is fine — it's fleet-wide).

3.2. Configure vzdump jobs

esh-pve and esh-pve-nas already have vzdump jobs hitting local dump dirs. Edit them per Phase 2.3.

sfsrv-ana has no existing vzdump jobs (per audit). Create one:

Web UI: Datacenter → Backup → Add:

Field	Value
Storage	pbs-ana
Schedule	03:00 (daily)
Selection	All
Retention	keep-daily 7, weekly 4, monthly 12, yearly 3
Compression	zstd
Mode	snapshot
Email	your address, notifications on failure

3.3. First backups

Fire manual runs from each hypervisor. ESH site backups cross-WAN to ANA — first full dumps will take a while. Schedule overnight if bandwidth is an issue during the day.

3.4. Verify on PBS-ANA side

Web UI on PBS: Datastore → backups → Content. Should show namespaces / groups for guests from all 4 hypervisors by hostname-ish prefixes (PBS uses VMID + node).

Phase 3 done-state

• All 3 hypervisors have working PBS backups
• sfsrv-ana specifically goes from ZERO backup coverage to full vzdump coverage (biggest single gain of this migration)

Phase 4 — Onboard nh3-pve

Target state: nh3-pve uses PBS-ANA as backup target despite the WAN hop.

Same process as Phase 3. The only extra considerations:

• Bandwidth: first full backups of NH3 VMs cross the WAN to ANA. Estimate size and run during off-hours. Incrementals after that should be small.
• Verify WAN stability: if the link flaps during a backup, PBS retries automatically but a multi-hour drop will fail. Don't start a 50 GB backup during known-unstable windows.

Phase 4 done-state

• All 5 hypervisors back up to PBS-ANA
• Verified: ~24 guests total across the fleet, all with recent PBS snapshots

Phase 5 — Provision PBS-NH3 (DR mirror)

Target state: PBS-NH3 running on nh3-pve, ready to pull from PBS-ANA.

5.1. Create Debian VM on nh3-pve

Same pattern as Phase 1.1:

Setting	Value
Name	pbs-nh3
VMID	next available on nh3-pve
CPU	2 cores
RAM	4 GB
Disk	32 GB OS + data disk (see 5.3)
Network	vmbr0

5.2. Install PBS

Same as Phase 1.2.

5.3. Local datastore

PBS-NH3 uses local storage rather than NFS (keeps the two PBSes on different failure domains). Two options:

Option A: attach a second virtual disk to the VM (e.g. 2 TB on nh3-pve's local-lvm or whatever storage is available), format ext4, mount at /mnt/pbs-datastore.

Option B: mount Synology Btrfs share via NFS or CIFS. Simpler storage admin but couples PBS-NH3 to the Synology's availability.

Recommended: Option A. Keeps it self-contained.

# Inside the PBS-NH3 VM, after attaching a disk /dev/sdb:
mkfs.ext4 /dev/sdb
mkdir -p /mnt/pbs-datastore
echo '/dev/sdb  /mnt/pbs-datastore  ext4  defaults  0  2' >> /etc/fstab
mount -a

5.4. Create datastore

Web UI: Datastore → Add, name backups-mirror, path /mnt/pbs-datastore.

Phase 5 done-state

• PBS-NH3 reachable, datastore ready
• No sync job yet — that's Phase 6

Phase 6 — Sync PBS-ANA → PBS-NH3

Target state: PBS-NH3 nightly pulls everything from PBS-ANA.

6.1. Create a remote entry on PBS-NH3

PBS-NH3 web UI: Configuration → Remotes → Add:

Field	Value
Remote ID	pbs-ana
Host	PBS-ANA IP
User ID	root@pam!fleet-vzdump (or a dedicated read-only token — see 6.3)
Password	token secret
Fingerprint	PBS-ANA's cert fingerprint

6.2. Create a sync job

Web UI: Datastore → backups-mirror → Sync Jobs → Add:

Field	Value
Remote	pbs-ana
Remote Datastore	backups
Sync Direction	Pull
Schedule	*-*-* 06:00:00 (after primary backups at 03:00)
Max Depth	unlimited
Owner	root@pam

Save. Run once manually to confirm.

First run transfers everything the primary has. This is one-time heavy — schedule on a weekend if bandwidth is a concern.

6.3. (Optional) tighten the token

The fleet-vzdump token has write permissions on PBS-ANA. For sync-only access, create a second read-only token on PBS-ANA (Datastore.Audit + Remote.Audit) and use that on the PBS-NH3 remote config. Reduces blast radius if PBS-NH3 ever compromised.

6.4. Set retention on PBS-NH3

Shorter than primary (it's DR, not archival):

Web UI: Datastore → backups-mirror → Prune & GC → Edit:

Field	Value
keep-last	7
keep-daily	7
keep-weekly	4

Phase 6 done-state

• Nightly pull from PBS-ANA → PBS-NH3 runs cleanly
• PBS-NH3 holds last 7 days of backups (shorter retention, DR focus)

Phase 7 — Burn-in

Target state: ~1 week of clean operation before retiring legacy targets.

Daily checks:

# PBS-ANA datastore growth is sensible
ssh root@pbs-ana 'df -h /mnt/pbs-datastore'

# PBS-NH3 sync job succeeded overnight
ssh root@pbs-nh3 'journalctl -u proxmox-backup-proxy --since yesterday | grep -i sync'

# Every hypervisor's last backup is ≤24h old
# (Run refresh-proxmox-info.sh and inspect the backup-coverage section)
scripts/refresh-proxmox-info.sh all

Watch for:

• Failed backup jobs (email alerts if enabled)
• Sync job duration growing unexpectedly (bandwidth issue)
• Datastore approaching capacity (bump retention or storage)
• VMs that don't dirty-bitmap incremental (missing QEMU Guest Agent, shut-down VMs always do full dumps)

Phase 8 — Retire legacy vzdump targets

Target state: no more old-style local-dump backups running; just PBS.

Only after Phase 7 passes:

• On each hypervisor: remove old vzdump jobs that targeted local dump dirs.
• Optionally delete old VMA files from local dump storage to reclaim space (before deleting, verify PBS has the same vintage).
• Update servers/*-pve/proxmox-details.txt by re-running scripts/refresh-proxmox-info.sh all — the backup coverage section should now show every guest covered by the PBS-ANA job.

Phase 9 — Update this workspace

• Update STATUS.md coverage table to show PBS deployment state.
• Update memory project_backup_pipeline_gaps.md — PBS closes several of the open gaps.
• Update README.md backup pipeline section to describe the new topology.
• Add an entry for PBS-ANA and PBS-NH3 in servers/ (with README + ssh-target). They're first-class infrastructure now.
• Homepage Infra-ANA card pointing at PBS-ANA Web UI; Infra-NH3 for PBS-NH3.

Rollback posture

At any phase, you can stop and leave the fleet in its current state:

• Phase 1 incomplete: no impact, PBS-ANA just sits idle.
• Phase 2+ incomplete: affected hypervisors still have their old vzdump jobs running. Until Phase 8, old targets are kept as redundant. Rolling back one hypervisor means deleting its PBS storage entry and re-enabling its old job.
• Phase 6+ incomplete: PBS-ANA still provides primary backup coverage. DR mirror just isn't built yet.