esh-pfi-infrastructure/docs/runbooks/pbs-deployment.md

Proxmox Backup Server deployment runbook

Stand up PBS-ANA as the fleet's primary vzdump target, then PBS-NH3 as a DR mirror that pulls nightly from PBS-ANA. Migrate all 5 hypervisors (pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) off their current local-dump vzdump targets onto PBS-ANA.

Architecture decision + rationale: see STATUS.md and the earlier session transcript. Short version: one primary at ANA, one-way sync to NH3, ANA owns retention policy.

Estimated total time: 4–6 hours spread across sittings. Each phase below is a self-contained work unit with a clear done-state; you can stop after any phase without leaving the fleet in a bad state.

Phase 0 — Prerequisites (one-time, before Phase 1)

0.1. Check current NFS exports

ssh -t 10.250.50.50 'sudo exportfs -v'

The existing broad exports (e.g. /mnt/backup → 10.0.0.0/8) cover pfi-pve but use root_squash — which will block PBS from writing its datastore metadata. We need a dedicated, no-squash export scoped to the PBS-ANA VM's IP.

0.2. Create a dedicated export for PBS-ANA

Assign PBS-ANA its LAN IP (let's say 10.250.50.90 — Phase 1.1 uses this), then on the Debian NAS:

ssh -t 10.250.50.50 '
  sudo mkdir -p /mnt/backup/pbs-ana &&
  sudo chown root:root /mnt/backup/pbs-ana &&
  sudo chmod 755 /mnt/backup/pbs-ana &&

  echo "/mnt/backup/pbs-ana  10.250.50.90(rw,sync,no_root_squash,no_subtree_check,sec=sys)" \
    | sudo tee /etc/exports.d/pbs-ana.exports > /dev/null &&

  sudo exportfs -ra &&
  sudo exportfs -v | grep pbs-ana
'

Should print an /mnt/backup/pbs-ana line with no_root_squash.

The existing cockpit-file-sharing.exports is managed by Cockpit's File Sharing module; keep the PBS-specific export in a separate file so Cockpit doesn't clobber it.

0.3. Decide datastore paths

• PBS-ANA datastore path: /mnt/backup/pbs-ana/ on the NAS, mounted at /mnt/pbs-datastore/ inside the PBS-ANA VM (Phase 1.3).
• PBS-NH3 datastore path: local-only on nh3-pve, mounted at /mnt/pbs-datastore/ inside the PBS-NH3 VM (Phase 5.3).

0.4. Side-note: /mnt/pve-VMStorage is exported to <world>

Not blocking PBS work but worth tightening during a Cockpit pass — reduce to 10.0.0.0/8 or more specific. Open item, not blocker.

0.5. ZFS case-insensitivity breaks NFSv4 writes

The Debian NAS's /mnt/backup is a ZFS dataset with casesensitivity=insensitive. ZFS + NFSv4 + case-insensitive dataset is a known-bad combo: lookups/readdir work, create/write ops fail with EACCES even for root with no_root_squash.

Workaround: mount with NFSv3 instead of NFSv4. Update the VM's /etc/fstab filesystem type to nfs and options to vers=3. PBS's access pattern (dedupe chunks are writen-once/read-many) works fine over NFSv3.

Example /etc/fstab line that works:

10.250.50.50:/mnt/backup/pbs-ana  /mnt/pbs-datastore  nfs  defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=3  0  0

Cleaner long-term fix (not required, but nicer): create a case-sensitive child dataset for PBS's data:

sudo zfs create -o casesensitivity=sensitive -o mountpoint=/mnt/backup/pbs-ds \
     NASPool/backupStore/pbs-ds

Then export/mount /mnt/backup/pbs-ds via NFSv4 as originally planned. Case-sensitivity can't be changed on an existing dataset, which is why the parent is stuck on case-insensitive.

Phase 1 — Provision PBS-ANA (VM on pfi-pve)

Target state: a fresh Debian 12 VM on pfi-pve running proxmox-backup-server, with a datastore configured on the NFS-mounted NAS path. Reachable at pbs-ana.phasefinal.com (or by IP).

1.1. Create a Debian 12 VM on pfi-pve

Via the pve web UI or CLI. Starting specs:

Setting	Value
Name	pbs-ana
VMID	next available (probably 114 given pfi-pve's fleet)
ISO	debian-12.x.x-amd64-netinst
CPU	4 cores
RAM	8 GB (can be tight at scale; bump if needed)
Disk	32 GB on local-lvm (for OS only — datastore is NFS)
Network	vmbr0, firewall on
Onboot	yes

Install Debian 12 minimal (no desktop, SSH server, standard system utilities).

Once up, set a static IP in the servers/ LAN range (e.g. 10.250.50.90) or DHCP-reserved for stability.

1.2. Install PBS package

SSH in as root:

# Add the PBS no-subscription repo
cat > /etc/apt/sources.list.d/pbs-no-subscription.list <<'EOF'
deb http://download.proxmox.com/debian/pbs bookworm pbs-no-subscription
EOF

wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg \
  -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg

apt update
apt full-upgrade -y
apt install -y proxmox-backup-server

Reboot once the PBS kernel is installed (PBS kernel is Proxmox's patched one, like PVE hosts):

systemctl reboot

After reboot, verify the web UI is reachable:

https://10.250.50.90:8007/

Default login: root / (the Debian root password you set at install).

1.3. Mount NAS datastore path

On the PBS VM:

# Create mount point
mkdir -p /mnt/pbs-datastore

# Add persistent NFS mount
cat >> /etc/fstab <<'EOF'
10.250.50.50:/mnt/backup/pbs-ana  /mnt/pbs-datastore  nfs4  defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=4.2  0  0
EOF

# Create the dir on NAS side if it doesn't exist
ssh ana-nas 'mkdir -p /mnt/backup/pbs-ana && chown root:root /mnt/backup/pbs-ana'

# Mount
mount -a
mount | grep pbs-datastore
# Should show the NFS mount active
df -h /mnt/pbs-datastore

1.4. Create PBS datastore

In the PBS web UI: Datastore → Add Datastore:

Field	Value
Name	backups
Backing Path	/mnt/pbs-datastore
Retention (can set later)	—
GC Schedule	Sunday 06:00
Prune Schedule	Daily 05:30

The first time PBS writes to the datastore it initializes .chunks/, .lock, etc. Takes a few seconds.

1.4b. Create per-hypervisor namespaces

VMIDs are NOT globally unique across PVE hosts (esh-pve and sfsrv-ana both have VM 100 for instance). Without namespaces, their backups collide under the same /vm/100/ path in the datastore. Create a namespace per hypervisor up front:

proxmox-backup-manager doesn't manage namespaces — they're created through the API/web UI or the proxmox-backup-client tool.

Easiest: web UI. Datastore → backups → Content → top of pane there's a namespace selector with an Add NS button. Add one per hypervisor.

Scripted via proxmox-backup-client (run on PBS-ANA):

export PBS_REPOSITORY='root@pam@localhost:backups'
export PBS_PASSWORD='<root-pam-password>'
for ns in pfi-pve nh3-pve esh-pve esh-pve-nas sfsrv-ana; do
  proxmox-backup-client namespace create "$ns"
done
proxmox-backup-client namespace list

Scripted via API (when ssh access is more convenient than shell on PBS-ANA):

TOKEN='<fleet-vzdump-secret>'
for ns in pfi-pve nh3-pve esh-pve esh-pve-nas sfsrv-ana; do
  curl -sk \
    -H "Authorization: PBSAPIToken=root@pam!fleet-vzdump:$TOKEN" \
    -X POST \
    https://10.250.50.90:8007/api2/json/admin/datastore/backups/namespace \
    -d "{\"ns\":\"$ns\"}"
done

Each PVE client later (Phase 2.1, 3.1, 4) sets its Namespace field to its own hostname when configuring the PBS storage.

1.4c. Create a verify job

New snapshots land unverified — PBS treats "backup completed" and "backup verified" as separate states. A verify job hash-checks chunk data (not just the manifest), flips snapshots to verified, and catches bitrot on aging data.

Web UI: Datastore → backups → Verify Jobs → Add:

Field	Value
Schedule	sat 23:00
Ignore verified snapshots	✓
Re-verify after (days)	30
Max depth	blank (unlimited)
Namespace	blank (root — recurses into all namespaces)
Comment	fleet verify — new + 30d re-check

Rationale for these defaults:

• sat 23:00 — sits between the daily 03:00 backup window and the Sunday 06:00 GC run, so verify never fights GC for I/O, and every week's new backups get verified before pruning decisions happen.
• ignore-verified: true + outdated-after: 30d — efficient steady state. First run after a backup night does the new snapshots only; a 30-day rolling re-verify catches silent chunk corruption.
• unlimited depth, root namespace — one job covers all 5 hypervisor namespaces. Split into per-namespace jobs only if you want per-hypervisor visibility into verify failures (not necessary for a fleet this size).

Verify runs are I/O-heavy on the datastore — on the NFS-backed PBS-ANA, expect a full-datastore verify to take hours once the datastore grows. The ignore-verified flag keeps incremental verify cheap; only the 30-day-aged portion is re-read each run.

1.5. Create an API token for hypervisors to use

PBS web UI: Configuration → Access Control → API Token → Add:

Field	Value
User	root@pam
Token Name	fleet-vzdump
Privilege Separation	off (so the token inherits root perms)

Save the token secret — this is the only time PBS shows it. Store in your password manager alongside the other fleet creds.

Token identifier looks like: root@pam!fleet-vzdump Secret looks like: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

1.6. Note PBS fingerprint

For client connections:

# On PBS VM
proxmox-backup-manager cert info | grep -i fingerprint

Save this fingerprint — each PVE client needs it to verify PBS's cert.

Phase 1 done-state

• PBS-ANA reachable at https://<ip>:8007/
• Datastore backups created and healthy
• API token root@pam!fleet-vzdump issued
• Cert fingerprint recorded

Phase 2 — Migrate pfi-pve's vzdump jobs onto PBS-ANA

Target state: pfi-pve sends its nightly backups to PBS-ANA.

2.1. Add PBS-ANA as storage on pfi-pve

Web UI: Datacenter → Storage → Add → Proxmox Backup Server:

Field	Value
ID	pbs-ana
Server	PBS-ANA IP
Username	root@pam!fleet-vzdump
Password	the token secret from 1.5
Datastore	backups
Namespace	pfi-pve (match the hypervisor name — see 1.4b)
Fingerprint	from 1.6
Content	VZDump backup file
Nodes	pve (pfi-pve)

Click Add. Storage should show as available within seconds.

Critical: every hypervisor gets its own namespace value matching its own name — nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana. Prevents VMID collisions (not-globally-unique VMIDs would otherwise land in the same /vm/<vmid>/ path).

2.2. Test backup of one VM

Pick a small VM (e.g. an idle LXC like ana-filebot, VMID 112). Web UI: select VM → Backup → Backup Now → Storage: pbs-ana.

First backup is a full dump. For a small LXC, a few minutes. Verify:

• Backup completes with green OK status
• In PBS UI: Datastore → backups shows the snapshot under ct/112/ or vm/112/

2.3. Migrate the scheduled backup job

Web UI: Datacenter → Backup. Edit the existing pfi-pve backup job (backup-4b911ca7-92df from the audit).

Change	From	To
Storage	naspool-vmstorage (was pve-truenas until 2026-09-06)	pbs-ana
Mode	snapshot	snapshot
Compression	zstd	zstd

Save. Next scheduled run (03:00 by default) fires against PBS-ANA.

2.4. Run it manually to verify

Web UI: backup job → Run Now. Watch each VM's backup complete. Larger VMs (PFI-Postgres 8GB RAM, PFI-Mongo, etc.) take 10-30 minutes each on first full backup. Subsequent backups will be minutes due to dirty-bitmap incrementals.

2.5. Keep legacy target for overlap

The old pve-truenas storage id was renamed naspool-vmstorage on 2026-09-06 (same path, /NASPool/pve-VMStorage, on the rebuilt raidz2 pool; legacy vzdump pruned to newest-per-guest). Historical note follows. Wait ~1 week of successful PBS runs before retiring it. Double-coverage period is cheap insurance.

Phase 2 done-state

• pfi-pve's 11 guests have first PBS snapshots
• Scheduled job uses PBS-ANA
• Old pve-truenas target (now naspool-vmstorage) still exists but no new jobs write to it

Phase 3 — Onboard esh-pve, esh-pve-nas, sfsrv-ana

Target state: all three remaining ANA/ESH hypervisors use PBS-ANA.

3.1. Repeat Phase 2.1 on each hypervisor

On each of esh-pve, esh-pve-nas, sfsrv-ana:

• Web UI: Datacenter → Storage → Add → Proxmox Backup Server (same fields as Phase 2.1; same token is fine — it's fleet-wide).

3.2. Configure vzdump jobs

esh-pve and esh-pve-nas already have vzdump jobs hitting local dump dirs. Edit them per Phase 2.3.

sfsrv-ana has no existing vzdump jobs (per audit). Create one:

Web UI: Datacenter → Backup → Add:

Field	Value
Storage	pbs-ana
Schedule	03:00 (daily)
Selection	All
Retention	keep-daily 7, weekly 4, monthly 12, yearly 3
Compression	zstd
Mode	snapshot
Email	your address, notifications on failure

3.3. First backups

Fire manual runs from each hypervisor. ESH site backups cross-WAN to ANA — first full dumps will take a while. Schedule overnight if bandwidth is an issue during the day.

3.4. Verify on PBS-ANA side

Web UI on PBS: Datastore → backups → Content. Should show namespaces / groups for guests from all 4 hypervisors by hostname-ish prefixes (PBS uses VMID + node).

Phase 3 done-state

• All 3 hypervisors have working PBS backups
• sfsrv-ana specifically goes from ZERO backup coverage to full vzdump coverage (biggest single gain of this migration)

Phase 4 — Onboard nh3-pve

Target state: nh3-pve uses PBS-ANA as backup target despite the WAN hop.

Same process as Phase 3. The only extra considerations:

• Bandwidth: first full backups of NH3 VMs cross the WAN to ANA. Estimate size and run during off-hours. Incrementals after that should be small.
• Verify WAN stability: if the link flaps during a backup, PBS retries automatically but a multi-hour drop will fail. Don't start a 50 GB backup during known-unstable windows.

Phase 4 done-state

• All 5 hypervisors back up to PBS-ANA
• Verified: ~24 guests total across the fleet, all with recent PBS snapshots

Phase 5 — Provision PBS-NH3 (DR mirror)

Target state: PBS-NH3 running on nh3-pve, ready to pull from PBS-ANA.

5.1. Create Debian VM on nh3-pve

Same pattern as Phase 1.1:

Setting	Value
Name	pbs-nh3
VMID	next available on nh3-pve
CPU	2 cores
RAM	4 GB
Disk	32 GB OS + data disk (see 5.3)
Network	vmbr0

5.2. Install PBS

Same as Phase 1.2.

5.3. Datastore backing — Synology NFS (chosen 2026-04-22)

PBS-NH3 mounts a Synology NFS share rather than using a local virtual disk. Simpler storage admin; tradeoff is that PBS-NH3 now shares its failure domain with the other NH3 backup paths (nh3-docker restic, nh3-dev restic, Backrest repos). Acceptable for DR purposes because PBS-ANA remains primary.

Synology-side export setup (DSM):

1. Create a dedicated share, e.g. pbs-nh3, on the target volume.

2. Control Panel → Shared Folder → [share] → Edit → NFS Permissions → Add/Edit:

   Field	Value
   Hostname/IP	PBS-NH3 VM IP
   Privilege	Read/Write
   Squash	No mapping (Synology's label for no_root_squash)
   Security	sys
   Enable asynchronous	on
   Allow connections from non-privileged ports	on

3. Verify no Advanced Permissions ACL denies root write — those override NFS perms and cause silent write failures.

Client-side mount (on PBS-NH3 VM):

Use NFSv3, not NFSv4 — see the gotcha below. Synology's "Advanced Permissions" layer an NFSv4 ACL on top of POSIX mode that is invisible to ls but denies writes to unprivileged users (including the backup uid-34 that PBS runs as), even when the directory mode is 777. Root bypasses this via no_root_squash, which is why a root touch succeeds but the datastore init fails.

mkdir -p /mnt/pbs-datastore

cat >> /etc/fstab <<'EOF'
10.100.50.50:/volume1/pbs-nh3  /mnt/pbs-datastore  nfs  defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=3  0  0
EOF

mount -a
df -h /mnt/pbs-datastore

# Two-layer smoke test: root AND the backup user that PBS runs as.
# The backup-user check is the one that actually matters — if root
# works but backup doesn't, you're hitting the ACL override.
touch /mnt/pbs-datastore/root-write-test && rm /mnt/pbs-datastore/root-write-test
sudo -u backup touch /mnt/pbs-datastore/backup-user-test && rm /mnt/pbs-datastore/backup-user-test

Ensure NFSv3 is reachable on the Synology side: Control Panel → File Services → NFS → Advanced — confirm "Minimum NFS Protocol" is 3 (not 4.0+). Default is usually 3; only an issue if someone hardened it previously.

Fallback if NFSv3 still fails the backup-user check: the deny is a Synology-local syno_acl (ls -la on the Synology shows POSIX mode d---------+ with a + for extended ACL). DSM "Enable Advanced Permissions" unchecked does NOT reset this once the archive flag has_ACL,is_support_ACL is set on the share.

Diagnose from the Synology shell:

sudo synoacltool -get /volume1/<share>
sudo ls -la /volume1/<share>/

A + after the permission string + a group:administrators:allow:... ACL entry + POSIX d--------- is the smoking gun: only members of the administrators group have access, which is why root (with no_root_squash) writes but uid-34 (backup) doesn't.

Fix (confirmed working 2026-04-22): keep Squash: No mapping (i.e. no_root_squash) AND flatten the share to Linux/POSIX mode with chmod 777. This drops the syno_acl entirely — verifiable by synoacltool -get returning "It's Linux mode" and ls -la showing drwxrwxrwx with NO trailing +.

On the Synology shell:

sudo chmod 777 /volume1/<share>

# Verify pure POSIX, no ACL
sudo synoacltool -get /volume1/<share>   # should say "It's Linux mode"
ls -la /volume1/<share>/                 # should show drwxrwxrwx (no '+')

Pure POSIX 777 is a cleaner long-term config than the ACL-grant approach — fewer permission-translation layers between NFSv3 and Btrfs, and no chance of ACL inheritance surprises on PBS-created subdirectories.

Why all_squash alone doesn't work: it lets backup-user writes through (because the ACL-granted admin gets the mapped uid), but breaks PBS's chown() during init. Squashed-admin doesn't have CAP_CHOWN on the Synology side → EPERM. Only real root (via no_root_squash) can chown to uid 34.

Why squash alone fails: even with all_squash + anonuid=1024 letting backup-user writes succeed (because admin is in administrators ACL), PBS's datastore-init calls chown on newly-created paths. Squashed-admin doesn't have CAP_CHOWN on the Synology side → EPERM. Only a real root (via no_root_squash) can chown to uid 34.

Smoke-test all three paths after the fix:

sudo touch /mnt/pbs-datastore/t && rm /mnt/pbs-datastore/t && echo ROOT_OK
sudo -u backup touch /mnt/pbs-datastore/t && rm /mnt/pbs-datastore/t && echo BACKUP_OK
touch /mnt/pbs-datastore/t && chown 34:34 /mnt/pbs-datastore/t && rm /mnt/pbs-datastore/t && echo CHOWN_OK

All three must pass before PBS will init cleanly.

Historical note: both PBS instances ended up on NFSv3 for unrelated reasons — the ANA side because of ZFS case-insensitivity, the NH3 side because of Synology ACL override. NFSv3 is the safer default for PBS-on-NFS regardless of backend.

5.4. Create datastore

Web UI: Datastore → Add, name backups-mirror, path /mnt/pbs-datastore.

5.5. Create a verify job on the mirror

Same pattern as Phase 1.4c on PBS-ANA — verification doesn't replicate across PBS instances, so the mirror needs its own job to catch bitrot on the local datastore disk.

Web UI: Datastore → backups-mirror → Verify Jobs → Add:

Field	Value
Schedule	sun 12:00
Ignore verified snapshots	✓
Re-verify after (days)	30
Max depth	blank (unlimited)
Namespace	blank
Comment	mirror verify — catches DR-side bitrot

Schedule sits after the 06:00 sync job finishes, so newly-synced snapshots get verified same day.

Phase 5 done-state

• PBS-NH3 reachable, datastore ready
• No sync job yet — that's Phase 6

Phase 6 — Sync PBS-ANA → PBS-NH3

Target state: PBS-NH3 nightly pulls everything from PBS-ANA.

6.1. Create a remote entry on PBS-NH3

PBS-NH3 web UI: Configuration → Remotes → Add:

Field	Value
Remote ID	pbs-ana
Host	PBS-ANA IP
User ID	root@pam!fleet-vzdump (or a dedicated read-only token — see 6.3)
Password	token secret
Fingerprint	PBS-ANA's cert fingerprint

6.2. Create a sync job

Web UI: Datastore → backups-mirror → Sync Jobs → Add:

Field	Value
Remote	pbs-ana
Remote Datastore	backups
Sync Direction	Pull
Schedule	*-*-* 06:00:00 (after primary backups at 03:00)
Max Depth	unlimited
Owner	root@pam

Save. Run once manually to confirm.

First run transfers everything the primary has. This is one-time heavy — schedule on a weekend if bandwidth is a concern.

6.3. (Optional) tighten the token

The fleet-vzdump token has write permissions on PBS-ANA. For sync-only access, create a second read-only token on PBS-ANA (Datastore.Audit + Remote.Audit) and use that on the PBS-NH3 remote config. Reduces blast radius if PBS-NH3 ever compromised.

6.4. Set retention on PBS-NH3

Shorter than primary (it's DR, not archival):

Web UI: Datastore → backups-mirror → Prune & GC → Edit:

Field	Value
keep-last	7
keep-daily	7
keep-weekly	4

Phase 6 done-state

• Nightly pull from PBS-ANA → PBS-NH3 runs cleanly
• PBS-NH3 holds last 7 days of backups (shorter retention, DR focus)

Phase 7 — Burn-in

Target state: ~1 week of clean operation before retiring legacy targets.

Daily checks:

# PBS-ANA datastore growth is sensible
ssh root@pbs-ana 'df -h /mnt/pbs-datastore'

# PBS-NH3 sync job succeeded overnight
ssh root@pbs-nh3 'journalctl -u proxmox-backup-proxy --since yesterday | grep -i sync'

# Every hypervisor's last backup is ≤24h old
# (Run refresh-proxmox-info.sh and inspect the backup-coverage section)
scripts/refresh-proxmox-info.sh all

Watch for:

• Failed backup jobs (email alerts if enabled)
• Sync job duration growing unexpectedly (bandwidth issue)
• Datastore approaching capacity (bump retention or storage)
• VMs that don't dirty-bitmap incremental (missing QEMU Guest Agent, shut-down VMs always do full dumps)

Phase 8 — Retire legacy vzdump targets

Target state: no more old-style local-dump backups running; just PBS.

Only after Phase 7 passes:

• On each hypervisor: remove old vzdump jobs that targeted local dump dirs.
• Optionally delete old VMA files from local dump storage to reclaim space (before deleting, verify PBS has the same vintage).
• Update servers/*-pve/proxmox-details.txt by re-running scripts/refresh-proxmox-info.sh all — the backup coverage section should now show every guest covered by the PBS-ANA job.

Phase 9 — Update this workspace

• Update STATUS.md coverage table to show PBS deployment state.
• Update memory project_backup_pipeline_gaps.md — PBS closes several of the open gaps.
• Update README.md backup pipeline section to describe the new topology.
• Add an entry for PBS-ANA and PBS-NH3 in servers/ (with README + ssh-target). They're first-class infrastructure now.
• Homepage Infra-ANA card pointing at PBS-ANA Web UI; Infra-NH3 for PBS-NH3.

Rollback posture

At any phase, you can stop and leave the fleet in its current state:

• Phase 1 incomplete: no impact, PBS-ANA just sits idle.
• Phase 2+ incomplete: affected hypervisors still have their old vzdump jobs running. Until Phase 8, old targets are kept as redundant. Rolling back one hypervisor means deleting its PBS storage entry and re-enabling its old job.
• Phase 6+ incomplete: PBS-ANA still provides primary backup coverage. DR mirror just isn't built yet.