vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: PBS deployment runbook (ANA primary + NH3 DR mirror)

Browse Source 
End-to-end runbook for standing up Proxmox Backup Server across the
fleet. Path A architecture: single primary at ANA, one-way sync to NH3
for disaster recovery. All 5 hypervisors (pfi-pve, nh3-pve, esh-pve,
esh-pve-nas, sfsrv-ana) migrate from local-dump vzdump to PBS-ANA.

Key decisions captured in the runbook:
  - PBS in a Debian VM (not LXC) for clean capability model.
  - PBS-ANA on pfi-pve, datastore via NFS from 10.250.50.50 —
    separates backup data from hypervisor boot disk.
  - PBS-NH3 on nh3-pve with local storage (independent failure
    domain from ANA).
  - Dedicated fleet-vzdump API token; read-only sync token for
    PBS-NH3's pull job.
  - sfsrv-ana specifically goes from zero backup coverage to full
    vzdump coverage in Phase 3.

9 phases, each self-contained with a done-state and rollback
posture. User can stop between phases without leaving the fleet in a
bad state.

STATUS.md: added item 6b tracking this deployment. Original item 6
(cross-site rsync) now scoped to restic-only since PBS handles the
VM-image cross-site redundancy directly.
This commit is contained in:
vh
2026-04-21 17:14:46 -07:00
parent 76a0768fdb
commit 0368ab732a
2 changed files with 491 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
STATUS.md
+15
View File
@@ -91,6 +91,21 @@ significant work lands — don't let it drift quietly.
(`/mnt/backup/restic/repo/ana/`) and NH3 Synology data dir. Planned (`/mnt/backup/restic/repo/ana/`) and NH3 Synology data dir. Planned
since initial rest-server setup; not built. Needs Synology SSH since initial rest-server setup; not built. Needs Synology SSH
access first (item 8). ~20 min once access is there. access first (item 8). ~20 min once access is there.
- Note: cross-site mirroring for the VM-image layer is being
addressed by the PBS deployment (item 6b) — this rsync is now
scoped to restic repos only.
6b. **PBS deployment across the fleet.** Runbook lives at
`docs/runbooks/pbs-deployment.md`. Primary at ANA (VM on pfi-pve,
NFS datastore on Debian NAS), DR mirror at NH3 (VM on nh3-pve,
local datastore). One-way sync from ANA → NH3 nightly. All 5
hypervisors (pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana)
migrate off local-dump vzdump jobs onto PBS.
- Closes: SureFire's zero-coverage gap (currently no vzdump at all on sfsrv-ana)
- Closes: "cross-site redundancy for VM images" scope item
- Provides: cross-fleet block-level dedupe, dirty-bitmap incrementals,
real safe-prune retention, per-host encryption keys
- Est. 4–6h spread across sittings. 9 phases in the runbook.
7. **SureFire tenant backup plan decision.** Three options documented in 7. **SureFire tenant backup plan decision.** Three options documented in
`servers/sfsrv-ana/README.md`: `servers/sfsrv-ana/README.md`:
docs/runbooks/pbs-deployment.md
+476
View File
@@ -0,0 +1,476 @@
# Proxmox Backup Server deployment runbook
Stand up PBS-ANA as the fleet's primary vzdump target, then PBS-NH3 as
a DR mirror that pulls nightly from PBS-ANA. Migrate all 5 hypervisors
(pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) off their current
local-dump vzdump targets onto PBS-ANA.
Architecture decision + rationale: see `STATUS.md` and the earlier
session transcript. Short version: one primary at ANA, one-way sync to
NH3, ANA owns retention policy.
**Estimated total time: 4–6 hours spread across sittings.** Each phase
below is a self-contained work unit with a clear done-state; you can
stop after any phase without leaving the fleet in a bad state.
## Phase 0 — Prerequisites (one-time, before Phase 1)
Check once:
```bash
# Debian NAS (10.250.50.50) is exporting to pfi-pve already for
# rest-server-ana's /mnt/backup. We need to add a sibling export for
# PBS-ANA's datastore. SSH to the NAS:
ssh ana-nas 'cat /etc/exports'
```
Make sure `/mnt/backup` export allows pfi-pve (10.250.250.31) to mount
read-write. If it does (which it should, per the existing
rest-server-ana setup), we're good.
Decide up front:
- **PBS-ANA datastore path:** recommend `/mnt/backup/pbs-ana/` (sibling
to `/mnt/backup/restic/`). Create the dir on the NAS with the user
that PBS will run as (root in the VM maps to root on NFS if
`no_root_squash` is set — matches the rest-server-ana pattern).
- **PBS-NH3 datastore path:** local-only on nh3-pve. Recommend a new
Proxmox dir storage on a data disk; `/mnt/pbs-nh3/datastore/`.
## Phase 1 — Provision PBS-ANA (VM on pfi-pve)
**Target state:** a fresh Debian 12 VM on pfi-pve running
`proxmox-backup-server`, with a datastore configured on the NFS-mounted
NAS path. Reachable at `pbs-ana.phasefinal.com` (or by IP).
### 1.1. Create a Debian 12 VM on pfi-pve
Via the pve web UI or CLI. Starting specs:
| Setting | Value |
|---|---|
| Name | `pbs-ana` |
| VMID | next available (probably 114 given pfi-pve's fleet) |
| ISO | debian-12.x.x-amd64-netinst |
| CPU | 4 cores |
| RAM | 8 GB (can be tight at scale; bump if needed) |
| Disk | 32 GB on `local-lvm` (for OS only — datastore is NFS) |
| Network | vmbr0, firewall on |
| Onboot | yes |
Install Debian 12 minimal (no desktop, SSH server, standard system utilities).
Once up, set a static IP in the `servers/` LAN range (e.g.
`10.250.50.90`) or DHCP-reserved for stability.
### 1.2. Install PBS package
SSH in as root:
```bash
# Add the PBS no-subscription repo
cat > /etc/apt/sources.list.d/pbs-no-subscription.list <<'EOF'
deb http://download.proxmox.com/debian/pbs bookworm pbs-no-subscription
EOF
wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg \
-O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg
apt update
apt full-upgrade -y
apt install -y proxmox-backup-server
```
Reboot once the PBS kernel is installed (PBS kernel is Proxmox's
patched one, like PVE hosts):
```bash
systemctl reboot
```
After reboot, verify the web UI is reachable:
```
https://10.250.50.90:8007/
```
Default login: `root` / (the Debian root password you set at install).
### 1.3. Mount NAS datastore path
On the PBS VM:
```bash
# Create mount point
mkdir -p /mnt/pbs-datastore
# Add persistent NFS mount
cat >> /etc/fstab <<'EOF'
10.250.50.50:/mnt/backup/pbs-ana /mnt/pbs-datastore nfs4 defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=4.2 0 0
EOF
# Create the dir on NAS side if it doesn't exist
ssh ana-nas 'mkdir -p /mnt/backup/pbs-ana && chown root:root /mnt/backup/pbs-ana'
# Mount
mount -a
mount | grep pbs-datastore
# Should show the NFS mount active
df -h /mnt/pbs-datastore
```
### 1.4. Create PBS datastore
In the PBS web UI: **Datastore → Add Datastore**:
| Field | Value |
|---|---|
| Name | `backups` |
| Backing Path | `/mnt/pbs-datastore` |
| Retention (can set later) | — |
| GC Schedule | Sunday 06:00 |
| Prune Schedule | Daily 05:30 |
The first time PBS writes to the datastore it initializes `.chunks/`,
`.lock`, etc. Takes a few seconds.
### 1.5. Create an API token for hypervisors to use
PBS web UI: **Configuration → Access Control → API Token → Add**:
| Field | Value |
|---|---|
| User | `root@pam` |
| Token Name | `fleet-vzdump` |
| Privilege Separation | **off** (so the token inherits root perms) |
Save the token secret — this is the only time PBS shows it. Store in
your password manager alongside the other fleet creds.
Token identifier looks like: `root@pam!fleet-vzdump`
Secret looks like: `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`
### 1.6. Note PBS fingerprint
For client connections:
```bash
# On PBS VM
proxmox-backup-manager cert info | grep -i fingerprint
```
Save this fingerprint — each PVE client needs it to verify PBS's cert.
### Phase 1 done-state
- PBS-ANA reachable at `https://<ip>:8007/`
- Datastore `backups` created and healthy
- API token `root@pam!fleet-vzdump` issued
- Cert fingerprint recorded
## Phase 2 — Migrate pfi-pve's vzdump jobs onto PBS-ANA
**Target state:** pfi-pve sends its nightly backups to PBS-ANA.
### 2.1. Add PBS-ANA as storage on pfi-pve
Web UI: **Datacenter → Storage → Add → Proxmox Backup Server**:
| Field | Value |
|---|---|
| ID | `pbs-ana` |
| Server | PBS-ANA IP |
| Username | `root@pam!fleet-vzdump` |
| Password | the token secret from 1.5 |
| Datastore | `backups` |
| Fingerprint | from 1.6 |
| Content | VZDump backup file |
| Nodes | `pve` (pfi-pve) |
Click Add. Storage should show as available within seconds.
### 2.2. Test backup of one VM
Pick a small VM (e.g. an idle LXC like ana-filebot, VMID 112). Web UI:
**select VM → Backup → Backup Now → Storage: pbs-ana**.
First backup is a full dump. For a small LXC, a few minutes. Verify:
- Backup completes with green OK status
- In PBS UI: **Datastore → backups** shows the snapshot under
`ct/112/` or `vm/112/`
### 2.3. Migrate the scheduled backup job
Web UI: **Datacenter → Backup**. Edit the existing pfi-pve backup job
(`backup-4b911ca7-92df` from the audit).
| Change | From | To |
|---|---|---|
| Storage | `pve-truenas` (or current) | `pbs-ana` |
| Mode | snapshot | snapshot |
| Compression | zstd | zstd |
Save. Next scheduled run (03:00 by default) fires against PBS-ANA.
### 2.4. Run it manually to verify
Web UI: **backup job → Run Now**. Watch each VM's backup complete.
Larger VMs (PFI-Postgres 8GB RAM, PFI-Mongo, etc.) take 10-30 minutes
each on first full backup. Subsequent backups will be minutes due to
dirty-bitmap incrementals.
### 2.5. Keep legacy target for overlap
**Don't delete the old `pve-truenas` backup storage yet.** Wait ~1 week
of successful PBS runs before retiring it. Double-coverage period is
cheap insurance.
### Phase 2 done-state
- pfi-pve's 11 guests have first PBS snapshots
- Scheduled job uses PBS-ANA
- Old `pve-truenas` target still exists (for safety) but no new jobs
write to it
## Phase 3 — Onboard esh-pve, esh-pve-nas, sfsrv-ana
**Target state:** all three remaining ANA/ESH hypervisors use PBS-ANA.
### 3.1. Repeat Phase 2.1 on each hypervisor
On each of `esh-pve`, `esh-pve-nas`, `sfsrv-ana`:
- Web UI: Datacenter → Storage → Add → Proxmox Backup Server (same
fields as Phase 2.1; same token is fine — it's fleet-wide).
### 3.2. Configure vzdump jobs
`esh-pve` and `esh-pve-nas` already have vzdump jobs hitting local
dump dirs. Edit them per Phase 2.3.
`sfsrv-ana` has **no existing vzdump jobs** (per audit). Create one:
Web UI: Datacenter → Backup → Add:
| Field | Value |
|---|---|
| Storage | `pbs-ana` |
| Schedule | `03:00` (daily) |
| Selection | All |
| Retention | keep-daily 7, weekly 4, monthly 12, yearly 3 |
| Compression | zstd |
| Mode | snapshot |
| Email | your address, notifications on failure |
### 3.3. First backups
Fire manual runs from each hypervisor. ESH site backups cross-WAN to
ANA — first full dumps will take a while. Schedule overnight if
bandwidth is an issue during the day.
### 3.4. Verify on PBS-ANA side
Web UI on PBS: **Datastore → backups → Content**. Should show
namespaces / groups for guests from all 4 hypervisors by hostname-ish
prefixes (PBS uses VMID + node).
### Phase 3 done-state
- All 3 hypervisors have working PBS backups
- sfsrv-ana specifically goes from ZERO backup coverage to full
vzdump coverage (biggest single gain of this migration)
## Phase 4 — Onboard nh3-pve
**Target state:** nh3-pve uses PBS-ANA as backup target despite the
WAN hop.
Same process as Phase 3. The only extra considerations:
- **Bandwidth:** first full backups of NH3 VMs cross the WAN to ANA.
Estimate size and run during off-hours. Incrementals after that
should be small.
- **Verify WAN stability:** if the link flaps during a backup, PBS
retries automatically but a multi-hour drop will fail. Don't start
a 50 GB backup during known-unstable windows.
### Phase 4 done-state
- All 5 hypervisors back up to PBS-ANA
- Verified: ~24 guests total across the fleet, all with recent
PBS snapshots
## Phase 5 — Provision PBS-NH3 (DR mirror)
**Target state:** PBS-NH3 running on nh3-pve, ready to pull from
PBS-ANA.
### 5.1. Create Debian VM on nh3-pve
Same pattern as Phase 1.1:
| Setting | Value |
|---|---|
| Name | `pbs-nh3` |
| VMID | next available on nh3-pve |
| CPU | 2 cores |
| RAM | 4 GB |
| Disk | 32 GB OS + data disk (see 5.3) |
| Network | vmbr0 |
### 5.2. Install PBS
Same as Phase 1.2.
### 5.3. Local datastore
PBS-NH3 uses local storage rather than NFS (keeps the two PBSes on
different failure domains). Two options:
**Option A:** attach a second virtual disk to the VM (e.g. 2 TB on
nh3-pve's `local-lvm` or whatever storage is available), format ext4,
mount at `/mnt/pbs-datastore`.
**Option B:** mount Synology Btrfs share via NFS or CIFS. Simpler
storage admin but couples PBS-NH3 to the Synology's availability.
Recommended: **Option A**. Keeps it self-contained.
```bash
# Inside the PBS-NH3 VM, after attaching a disk /dev/sdb:
mkfs.ext4 /dev/sdb
mkdir -p /mnt/pbs-datastore
echo '/dev/sdb /mnt/pbs-datastore ext4 defaults 0 2' >> /etc/fstab
mount -a
```
### 5.4. Create datastore
Web UI: Datastore → Add, name `backups-mirror`, path `/mnt/pbs-datastore`.
### Phase 5 done-state
- PBS-NH3 reachable, datastore ready
- No sync job yet — that's Phase 6
## Phase 6 — Sync PBS-ANA → PBS-NH3
**Target state:** PBS-NH3 nightly pulls everything from PBS-ANA.
### 6.1. Create a remote entry on PBS-NH3
PBS-NH3 web UI: **Configuration → Remotes → Add**:
| Field | Value |
|---|---|
| Remote ID | `pbs-ana` |
| Host | PBS-ANA IP |
| User ID | `root@pam!fleet-vzdump` (or a dedicated read-only token — see 6.3) |
| Password | token secret |
| Fingerprint | PBS-ANA's cert fingerprint |
### 6.2. Create a sync job
Web UI: **Datastore → backups-mirror → Sync Jobs → Add**:
| Field | Value |
|---|---|
| Remote | `pbs-ana` |
| Remote Datastore | `backups` |
| Sync Direction | Pull |
| Schedule | `*-*-* 06:00:00` (after primary backups at 03:00) |
| Max Depth | unlimited |
| Owner | `root@pam` |
Save. Run once manually to confirm.
First run transfers everything the primary has. This is one-time heavy
— schedule on a weekend if bandwidth is a concern.
### 6.3. (Optional) tighten the token
The `fleet-vzdump` token has write permissions on PBS-ANA. For
sync-only access, create a second read-only token on PBS-ANA
(`Datastore.Audit` + `Remote.Audit`) and use that on the PBS-NH3
remote config. Reduces blast radius if PBS-NH3 ever compromised.
### 6.4. Set retention on PBS-NH3
Shorter than primary (it's DR, not archival):
Web UI: Datastore → backups-mirror → Prune & GC → Edit:
| Field | Value |
|---|---|
| keep-last | 7 |
| keep-daily | 7 |
| keep-weekly | 4 |
### Phase 6 done-state
- Nightly pull from PBS-ANA → PBS-NH3 runs cleanly
- PBS-NH3 holds last 7 days of backups (shorter retention, DR focus)
## Phase 7 — Burn-in
**Target state:** ~1 week of clean operation before retiring legacy targets.
Daily checks:
```bash
# PBS-ANA datastore growth is sensible
ssh root@pbs-ana 'df -h /mnt/pbs-datastore'
# PBS-NH3 sync job succeeded overnight
ssh root@pbs-nh3 'journalctl -u proxmox-backup-proxy --since yesterday | grep -i sync'
# Every hypervisor's last backup is ≤24h old
# (Run refresh-proxmox-info.sh and inspect the backup-coverage section)
scripts/refresh-proxmox-info.sh all
```
Watch for:
- Failed backup jobs (email alerts if enabled)
- Sync job duration growing unexpectedly (bandwidth issue)
- Datastore approaching capacity (bump retention or storage)
- VMs that don't dirty-bitmap incremental (missing QEMU Guest Agent,
shut-down VMs always do full dumps)
## Phase 8 — Retire legacy vzdump targets
**Target state:** no more old-style local-dump backups running; just PBS.
Only after Phase 7 passes:
- On each hypervisor: remove old vzdump jobs that targeted local dump
dirs.
- Optionally delete old VMA files from local dump storage to reclaim
space (before deleting, verify PBS has the same vintage).
- Update `servers/*-pve/proxmox-details.txt` by re-running
`scripts/refresh-proxmox-info.sh all` — the `backup coverage` section
should now show every guest covered by the PBS-ANA job.
## Phase 9 — Update this workspace
- Update `STATUS.md` coverage table to show PBS deployment state.
- Update memory `project_backup_pipeline_gaps.md` — PBS closes several
of the open gaps.
- Update `README.md` backup pipeline section to describe the new
topology.
- Add an entry for PBS-ANA and PBS-NH3 in `servers/` (with README +
ssh-target). They're first-class infrastructure now.
- Homepage Infra-ANA card pointing at PBS-ANA Web UI; Infra-NH3 for
PBS-NH3.
## Rollback posture
At any phase, you can stop and leave the fleet in its current state:
- Phase 1 incomplete: no impact, PBS-ANA just sits idle.
- Phase 2+ incomplete: affected hypervisors still have their old
vzdump jobs running. Until Phase 8, old targets are kept as
redundant. Rolling back one hypervisor means deleting its
PBS storage entry and re-enabling its old job.
- Phase 6+ incomplete: PBS-ANA still provides primary backup coverage.
DR mirror just isn't built yet.