esh-pfi-infrastructure/docs/runbooks/pbs-deployment.md

# Proxmox Backup Server deployment runbook

Stand up PBS-ANA as the fleet's primary vzdump target, then PBS-NH3 as
a DR mirror that pulls nightly from PBS-ANA. Migrate all 5 hypervisors
(pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) off their current
local-dump vzdump targets onto PBS-ANA.

Architecture decision + rationale: see `STATUS.md` and the earlier
session transcript. Short version: one primary at ANA, one-way sync to
NH3, ANA owns retention policy.

**Estimated total time: 4–6 hours spread across sittings.** Each phase
below is a self-contained work unit with a clear done-state; you can
stop after any phase without leaving the fleet in a bad state.

## Phase 0 — Prerequisites (one-time, before Phase 1)

### 0.1. Check current NFS exports

```bash
ssh -t 10.250.50.50 'sudo exportfs -v'
```

The existing broad exports (e.g. `/mnt/backup` → `10.0.0.0/8`) cover
pfi-pve but use `root_squash` — which will **block** PBS from writing
its datastore metadata. We need a dedicated, no-squash export scoped
to the PBS-ANA VM's IP.

### 0.2. Create a dedicated export for PBS-ANA

Assign PBS-ANA its LAN IP (let's say `10.250.50.90` — Phase 1.1 uses
this), then on the Debian NAS:

```bash
ssh -t 10.250.50.50 '
  sudo mkdir -p /mnt/backup/pbs-ana &&
  sudo chown root:root /mnt/backup/pbs-ana &&
  sudo chmod 755 /mnt/backup/pbs-ana &&

  echo "/mnt/backup/pbs-ana  10.250.50.90(rw,sync,no_root_squash,no_subtree_check,sec=sys)" \
    | sudo tee /etc/exports.d/pbs-ana.exports > /dev/null &&

  sudo exportfs -ra &&
  sudo exportfs -v | grep pbs-ana
'
```

Should print an `/mnt/backup/pbs-ana` line with `no_root_squash`.

The existing `cockpit-file-sharing.exports` is managed by Cockpit's
File Sharing module; keep the PBS-specific export in a separate file
so Cockpit doesn't clobber it.

### 0.3. Decide datastore paths

- **PBS-ANA datastore path:** `/mnt/backup/pbs-ana/` on the NAS, mounted
  at `/mnt/pbs-datastore/` inside the PBS-ANA VM (Phase 1.3).
- **PBS-NH3 datastore path:** local-only on nh3-pve, mounted at
  `/mnt/pbs-datastore/` inside the PBS-NH3 VM (Phase 5.3).

### 0.4. Side-note: `/mnt/pve-VMStorage` is exported to `<world>`

Not blocking PBS work but worth tightening during a Cockpit pass —
reduce to `10.0.0.0/8` or more specific. Open item, not blocker.

### 0.5. ZFS case-insensitivity breaks NFSv4 writes

The Debian NAS's `/mnt/backup` is a ZFS dataset with
`casesensitivity=insensitive`. ZFS + NFSv4 + case-insensitive dataset
is a known-bad combo: lookups/readdir work, `create`/`write` ops fail
with EACCES even for root with `no_root_squash`.

**Workaround:** mount with NFSv3 instead of NFSv4. Update the VM's
`/etc/fstab` filesystem type to `nfs` and options to `vers=3`. PBS's
access pattern (dedupe chunks are writen-once/read-many) works fine
over NFSv3.

Example `/etc/fstab` line that works:

```
10.250.50.50:/mnt/backup/pbs-ana  /mnt/pbs-datastore  nfs  defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=3  0  0
```

**Cleaner long-term fix** (not required, but nicer): create a
case-sensitive child dataset for PBS's data:

```
sudo zfs create -o casesensitivity=sensitive -o mountpoint=/mnt/backup/pbs-ds \
     NASPool/backupStore/pbs-ds
```

Then export/mount `/mnt/backup/pbs-ds` via NFSv4 as originally planned.
Case-sensitivity can't be changed on an existing dataset, which is why
the parent is stuck on case-insensitive.

## Phase 1 — Provision PBS-ANA (VM on pfi-pve)

**Target state:** a fresh Debian 12 VM on pfi-pve running
`proxmox-backup-server`, with a datastore configured on the NFS-mounted
NAS path. Reachable at `pbs-ana.phasefinal.com` (or by IP).

### 1.1. Create a Debian 12 VM on pfi-pve

Via the pve web UI or CLI. Starting specs:

| Setting | Value |
|---|---|
| Name | `pbs-ana` |
| VMID | next available (probably 114 given pfi-pve's fleet) |
| ISO | debian-12.x.x-amd64-netinst |
| CPU | 4 cores |
| RAM | 8 GB (can be tight at scale; bump if needed) |
| Disk | 32 GB on `local-lvm` (for OS only — datastore is NFS) |
| Network | vmbr0, firewall on |
| Onboot | yes |

Install Debian 12 minimal (no desktop, SSH server, standard system utilities).

Once up, set a static IP in the `servers/` LAN range (e.g.
`10.250.50.90`) or DHCP-reserved for stability.

### 1.2. Install PBS package

SSH in as root:

```bash
# Add the PBS no-subscription repo
cat > /etc/apt/sources.list.d/pbs-no-subscription.list <<'EOF'
deb http://download.proxmox.com/debian/pbs bookworm pbs-no-subscription
EOF

wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg \
  -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg

apt update
apt full-upgrade -y
apt install -y proxmox-backup-server
```

Reboot once the PBS kernel is installed (PBS kernel is Proxmox's
patched one, like PVE hosts):

```bash
systemctl reboot
```

After reboot, verify the web UI is reachable:

```
https://10.250.50.90:8007/
```

Default login: `root` / (the Debian root password you set at install).

### 1.3. Mount NAS datastore path

On the PBS VM:

```bash
# Create mount point
mkdir -p /mnt/pbs-datastore

# Add persistent NFS mount
cat >> /etc/fstab <<'EOF'
10.250.50.50:/mnt/backup/pbs-ana  /mnt/pbs-datastore  nfs4  defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=4.2  0  0
EOF

# Create the dir on NAS side if it doesn't exist
ssh ana-nas 'mkdir -p /mnt/backup/pbs-ana && chown root:root /mnt/backup/pbs-ana'

# Mount
mount -a
mount | grep pbs-datastore
# Should show the NFS mount active
df -h /mnt/pbs-datastore
```

### 1.4. Create PBS datastore

In the PBS web UI: **Datastore → Add Datastore**:

| Field | Value |
|---|---|
| Name | `backups` |
| Backing Path | `/mnt/pbs-datastore` |
| Retention (can set later) | — |
| GC Schedule | Sunday 06:00 |
| Prune Schedule | Daily 05:30 |

The first time PBS writes to the datastore it initializes `.chunks/`,
`.lock`, etc. Takes a few seconds.

### 1.4b. Create per-hypervisor namespaces

VMIDs are NOT globally unique across PVE hosts (esh-pve and sfsrv-ana
both have VM 100 for instance). Without namespaces, their backups
collide under the same `/vm/100/` path in the datastore. Create a
namespace per hypervisor up front:

`proxmox-backup-manager` doesn't manage namespaces — they're created
through the API/web UI or the `proxmox-backup-client` tool.

**Easiest: web UI.** Datastore → `backups` → Content → top of pane
there's a namespace selector with an **Add NS** button. Add one per
hypervisor.

**Scripted via `proxmox-backup-client`** (run on PBS-ANA):

```bash
export PBS_REPOSITORY='root@pam@localhost:backups'
export PBS_PASSWORD='<root-pam-password>'
for ns in pfi-pve nh3-pve esh-pve esh-pve-nas sfsrv-ana; do
  proxmox-backup-client namespace create "$ns"
done
proxmox-backup-client namespace list
```

**Scripted via API** (when ssh access is more convenient than shell
on PBS-ANA):

```bash
TOKEN='<fleet-vzdump-secret>'
for ns in pfi-pve nh3-pve esh-pve esh-pve-nas sfsrv-ana; do
  curl -sk \
    -H "Authorization: PBSAPIToken=root@pam!fleet-vzdump:$TOKEN" \
    -X POST \
    https://10.250.50.90:8007/api2/json/admin/datastore/backups/namespace \
    -d "{\"ns\":\"$ns\"}"
done
```

Each PVE client later (Phase 2.1, 3.1, 4) sets its Namespace field to
its own hostname when configuring the PBS storage.

### 1.4c. Create a verify job

New snapshots land unverified — PBS treats "backup completed" and
"backup verified" as separate states. A verify job hash-checks chunk
data (not just the manifest), flips snapshots to verified, and catches
bitrot on aging data.

Web UI: **Datastore → backups → Verify Jobs → Add**:

| Field | Value |
|---|---|
| Schedule | `sat 23:00` |
| Ignore verified snapshots | ✓ |
| Re-verify after (days) | `30` |
| Max depth | blank (unlimited) |
| Namespace | blank (root — recurses into all namespaces) |
| Comment | `fleet verify — new + 30d re-check` |

Rationale for these defaults:

- **`sat 23:00`** — sits between the daily 03:00 backup window and the
  Sunday 06:00 GC run, so verify never fights GC for I/O, and every
  week's new backups get verified before pruning decisions happen.
- **`ignore-verified: true`** + **`outdated-after: 30d`** — efficient
  steady state. First run after a backup night does the new snapshots
  only; a 30-day rolling re-verify catches silent chunk corruption.
- **unlimited depth, root namespace** — one job covers all 5
  hypervisor namespaces. Split into per-namespace jobs only if you
  want per-hypervisor visibility into verify failures (not necessary
  for a fleet this size).

Verify runs are I/O-heavy on the datastore — on the NFS-backed
PBS-ANA, expect a full-datastore verify to take hours once the
datastore grows. The `ignore-verified` flag keeps incremental verify
cheap; only the 30-day-aged portion is re-read each run.

### 1.5. Create an API token for hypervisors to use

PBS web UI: **Configuration → Access Control → API Token → Add**:

| Field | Value |
|---|---|
| User | `root@pam` |
| Token Name | `fleet-vzdump` |
| Privilege Separation | **off** (so the token inherits root perms) |

Save the token secret — this is the only time PBS shows it. Store in
your password manager alongside the other fleet creds.

Token identifier looks like: `root@pam!fleet-vzdump`
Secret looks like: `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`

### 1.6. Note PBS fingerprint

For client connections:

```bash
# On PBS VM
proxmox-backup-manager cert info | grep -i fingerprint
```

Save this fingerprint — each PVE client needs it to verify PBS's cert.

### Phase 1 done-state

- PBS-ANA reachable at `https://<ip>:8007/`
- Datastore `backups` created and healthy
- API token `root@pam!fleet-vzdump` issued
- Cert fingerprint recorded

## Phase 2 — Migrate pfi-pve's vzdump jobs onto PBS-ANA

**Target state:** pfi-pve sends its nightly backups to PBS-ANA.

### 2.1. Add PBS-ANA as storage on pfi-pve

Web UI: **Datacenter → Storage → Add → Proxmox Backup Server**:

| Field | Value |
|---|---|
| ID | `pbs-ana` |
| Server | PBS-ANA IP |
| Username | `root@pam!fleet-vzdump` |
| Password | the token secret from 1.5 |
| Datastore | `backups` |
| **Namespace** | **`pfi-pve`** (match the hypervisor name — see 1.4b) |
| Fingerprint | from 1.6 |
| Content | VZDump backup file |
| Nodes | `pve` (pfi-pve) |

Click Add. Storage should show as available within seconds.

> **Critical:** every hypervisor gets its own namespace value matching
> its own name — `nh3-pve`, `esh-pve`, `esh-pve-nas`, `sfsrv-ana`.
> Prevents VMID collisions (not-globally-unique VMIDs would otherwise
> land in the same `/vm/<vmid>/` path).

### 2.2. Test backup of one VM

Pick a small VM (e.g. an idle LXC like ana-filebot, VMID 112). Web UI:
**select VM → Backup → Backup Now → Storage: pbs-ana**.

First backup is a full dump. For a small LXC, a few minutes. Verify:
- Backup completes with green OK status
- In PBS UI: **Datastore → backups** shows the snapshot under
  `ct/112/` or `vm/112/`

### 2.3. Migrate the scheduled backup job

Web UI: **Datacenter → Backup**. Edit the existing pfi-pve backup job
(`backup-4b911ca7-92df` from the audit).

| Change | From | To |
|---|---|---|
| Storage | `naspool-vmstorage` (was `pve-truenas` until 2026-09-06) | `pbs-ana` |
| Mode | snapshot | snapshot |
| Compression | zstd | zstd |

Save. Next scheduled run (03:00 by default) fires against PBS-ANA.

### 2.4. Run it manually to verify

Web UI: **backup job → Run Now**. Watch each VM's backup complete.
Larger VMs (PFI-Postgres 8GB RAM, PFI-Mongo, etc.) take 10-30 minutes
each on first full backup. Subsequent backups will be minutes due to
dirty-bitmap incrementals.

### 2.5. Keep legacy target for overlap

**The old `pve-truenas` storage id was renamed `naspool-vmstorage` on 2026-09-06** (same path, `/NASPool/pve-VMStorage`, on the rebuilt raidz2 pool; legacy vzdump pruned to newest-per-guest). Historical note follows. Wait ~1 week
of successful PBS runs before retiring it. Double-coverage period is
cheap insurance.

### Phase 2 done-state

- pfi-pve's 11 guests have first PBS snapshots
- Scheduled job uses PBS-ANA
- Old `pve-truenas` target (now `naspool-vmstorage`) still exists but no new jobs
  write to it

## Phase 3 — Onboard esh-pve, esh-pve-nas, sfsrv-ana

**Target state:** all three remaining ANA/ESH hypervisors use PBS-ANA.

### 3.1. Repeat Phase 2.1 on each hypervisor

On each of `esh-pve`, `esh-pve-nas`, `sfsrv-ana`:

- Web UI: Datacenter → Storage → Add → Proxmox Backup Server (same
  fields as Phase 2.1; same token is fine — it's fleet-wide).

### 3.2. Configure vzdump jobs

`esh-pve` and `esh-pve-nas` already have vzdump jobs hitting local
dump dirs. Edit them per Phase 2.3.

`sfsrv-ana` has **no existing vzdump jobs** (per audit). Create one:

Web UI: Datacenter → Backup → Add:

| Field | Value |
|---|---|
| Storage | `pbs-ana` |
| Schedule | `03:00` (daily) |
| Selection | All |
| Retention | keep-daily 7, weekly 4, monthly 12, yearly 3 |
| Compression | zstd |
| Mode | snapshot |
| Email | your address, notifications on failure |

### 3.3. First backups

Fire manual runs from each hypervisor. ESH site backups cross-WAN to
ANA — first full dumps will take a while. Schedule overnight if
bandwidth is an issue during the day.

### 3.4. Verify on PBS-ANA side

Web UI on PBS: **Datastore → backups → Content**. Should show
namespaces / groups for guests from all 4 hypervisors by hostname-ish
prefixes (PBS uses VMID + node).

### Phase 3 done-state

- All 3 hypervisors have working PBS backups
- sfsrv-ana specifically goes from ZERO backup coverage to full
  vzdump coverage (biggest single gain of this migration)

## Phase 4 — Onboard nh3-pve

**Target state:** nh3-pve uses PBS-ANA as backup target despite the
WAN hop.

Same process as Phase 3. The only extra considerations:

- **Bandwidth:** first full backups of NH3 VMs cross the WAN to ANA.
  Estimate size and run during off-hours. Incrementals after that
  should be small.
- **Verify WAN stability:** if the link flaps during a backup, PBS
  retries automatically but a multi-hour drop will fail. Don't start
  a 50 GB backup during known-unstable windows.

### Phase 4 done-state

- All 5 hypervisors back up to PBS-ANA
- Verified: ~24 guests total across the fleet, all with recent
  PBS snapshots

## Phase 5 — Provision PBS-NH3 (DR mirror)

**Target state:** PBS-NH3 running on nh3-pve, ready to pull from
PBS-ANA.

### 5.1. Create Debian VM on nh3-pve

Same pattern as Phase 1.1:

| Setting | Value |
|---|---|
| Name | `pbs-nh3` |
| VMID | next available on nh3-pve |
| CPU | 2 cores |
| RAM | 4 GB |
| Disk | 32 GB OS + data disk (see 5.3) |
| Network | vmbr0 |

### 5.2. Install PBS

Same as Phase 1.2.

### 5.3. Datastore backing — Synology NFS (chosen 2026-04-22)

PBS-NH3 mounts a Synology NFS share rather than using a local virtual
disk. Simpler storage admin; tradeoff is that PBS-NH3 now shares its
failure domain with the other NH3 backup paths (nh3-docker restic,
nh3-dev restic, Backrest repos). Acceptable for DR purposes because
PBS-ANA remains primary.

**Synology-side export setup (DSM):**

1. Create a dedicated share, e.g. `pbs-nh3`, on the target volume.
2. **Control Panel → Shared Folder → [share] → Edit → NFS Permissions → Add/Edit:**
   | Field | Value |
   |---|---|
   | Hostname/IP | PBS-NH3 VM IP |
   | Privilege | Read/Write |
   | Squash | **No mapping** (Synology's label for `no_root_squash`) |
   | Security | sys |
   | Enable asynchronous | on |
   | Allow connections from non-privileged ports | on |

3. Verify no **Advanced Permissions** ACL denies `root` write — those
   override NFS perms and cause silent write failures.

**Client-side mount (on PBS-NH3 VM):**

Use **NFSv3**, not NFSv4 — see the gotcha below. Synology's "Advanced
Permissions" layer an NFSv4 ACL on top of POSIX mode that is invisible
to `ls` but denies writes to unprivileged users (including the
`backup` uid-34 that PBS runs as), even when the directory mode is
777. Root bypasses this via `no_root_squash`, which is why a root
`touch` succeeds but the datastore init fails.

```bash
mkdir -p /mnt/pbs-datastore

cat >> /etc/fstab <<'EOF'
10.100.50.50:/volume1/pbs-nh3  /mnt/pbs-datastore  nfs  defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=3  0  0
EOF

mount -a
df -h /mnt/pbs-datastore

# Two-layer smoke test: root AND the backup user that PBS runs as.
# The backup-user check is the one that actually matters — if root
# works but backup doesn't, you're hitting the ACL override.
touch /mnt/pbs-datastore/root-write-test && rm /mnt/pbs-datastore/root-write-test
sudo -u backup touch /mnt/pbs-datastore/backup-user-test && rm /mnt/pbs-datastore/backup-user-test
```

Ensure NFSv3 is reachable on the Synology side: **Control Panel →
File Services → NFS → Advanced** — confirm "Minimum NFS Protocol" is
3 (not 4.0+). Default is usually 3; only an issue if someone
hardened it previously.

**Fallback if NFSv3 still fails the backup-user check:** the deny is
a Synology-local syno_acl (`ls -la` on the Synology shows POSIX mode
`d---------+` with a `+` for extended ACL). DSM "Enable Advanced
Permissions" unchecked does NOT reset this once the archive flag
`has_ACL,is_support_ACL` is set on the share.

Diagnose from the Synology shell:

```
sudo synoacltool -get /volume1/<share>
sudo ls -la /volume1/<share>/
```

A `+` after the permission string + a `group:administrators:allow:...`
ACL entry + POSIX `d---------` is the smoking gun: only members of
the `administrators` group have access, which is why root (with
`no_root_squash`) writes but uid-34 (backup) doesn't.

**Fix (confirmed working 2026-04-22):** keep **Squash: `No mapping`**
(i.e. `no_root_squash`) AND flatten the share to Linux/POSIX mode
with `chmod 777`. This drops the syno_acl entirely — verifiable by
`synoacltool -get` returning "It's Linux mode" and `ls -la`
showing `drwxrwxrwx` with NO trailing `+`.

On the Synology shell:

```bash
sudo chmod 777 /volume1/<share>

# Verify pure POSIX, no ACL
sudo synoacltool -get /volume1/<share>   # should say "It's Linux mode"
ls -la /volume1/<share>/                 # should show drwxrwxrwx (no '+')
```

Pure POSIX 777 is a cleaner long-term config than the ACL-grant
approach — fewer permission-translation layers between NFSv3 and
Btrfs, and no chance of ACL inheritance surprises on PBS-created
subdirectories.

**Why `all_squash` alone doesn't work:** it lets backup-user writes
through (because the ACL-granted admin gets the mapped uid), but
breaks PBS's `chown()` during init. Squashed-admin doesn't have
CAP_CHOWN on the Synology side → EPERM. Only real root (via
`no_root_squash`) can chown to uid 34.

**Why squash alone fails:** even with `all_squash + anonuid=1024`
letting backup-user writes succeed (because admin is in
`administrators` ACL), PBS's datastore-init calls `chown` on
newly-created paths. Squashed-admin doesn't have CAP_CHOWN on the
Synology side → EPERM. Only a real root (via `no_root_squash`) can
chown to uid 34.

**Smoke-test all three paths after the fix:**

```bash
sudo touch /mnt/pbs-datastore/t && rm /mnt/pbs-datastore/t && echo ROOT_OK
sudo -u backup touch /mnt/pbs-datastore/t && rm /mnt/pbs-datastore/t && echo BACKUP_OK
touch /mnt/pbs-datastore/t && chown 34:34 /mnt/pbs-datastore/t && rm /mnt/pbs-datastore/t && echo CHOWN_OK
```

All three must pass before PBS will init cleanly.

**Historical note:** both PBS instances ended up on NFSv3 for
unrelated reasons — the ANA side because of ZFS case-insensitivity,
the NH3 side because of Synology ACL override. NFSv3 is the safer
default for PBS-on-NFS regardless of backend.

### 5.4. Create datastore

Web UI: Datastore → Add, name `backups-mirror`, path `/mnt/pbs-datastore`.

### 5.5. Create a verify job on the mirror

Same pattern as Phase 1.4c on PBS-ANA — verification doesn't replicate
across PBS instances, so the mirror needs its own job to catch bitrot
on the local datastore disk.

Web UI: **Datastore → backups-mirror → Verify Jobs → Add**:

| Field | Value |
|---|---|
| Schedule | `sun 12:00` |
| Ignore verified snapshots | ✓ |
| Re-verify after (days) | `30` |
| Max depth | blank (unlimited) |
| Namespace | blank |
| Comment | `mirror verify — catches DR-side bitrot` |

Schedule sits after the 06:00 sync job finishes, so newly-synced
snapshots get verified same day.

### Phase 5 done-state

- PBS-NH3 reachable, datastore ready
- No sync job yet — that's Phase 6

## Phase 6 — Sync PBS-ANA → PBS-NH3

**Target state:** PBS-NH3 nightly pulls everything from PBS-ANA.

### 6.1. Create a remote entry on PBS-NH3

PBS-NH3 web UI: **Configuration → Remotes → Add**:

| Field | Value |
|---|---|
| Remote ID | `pbs-ana` |
| Host | PBS-ANA IP |
| User ID | `root@pam!fleet-vzdump` (or a dedicated read-only token — see 6.3) |
| Password | token secret |
| Fingerprint | PBS-ANA's cert fingerprint |

### 6.2. Create a sync job

Web UI: **Datastore → backups-mirror → Sync Jobs → Add**:

| Field | Value |
|---|---|
| Remote | `pbs-ana` |
| Remote Datastore | `backups` |
| Sync Direction | Pull |
| Schedule | `*-*-* 06:00:00` (after primary backups at 03:00) |
| Max Depth | unlimited |
| Owner | `root@pam` |

Save. Run once manually to confirm.

First run transfers everything the primary has. This is one-time heavy
— schedule on a weekend if bandwidth is a concern.

### 6.3. (Optional) tighten the token

The `fleet-vzdump` token has write permissions on PBS-ANA. For
sync-only access, create a second read-only token on PBS-ANA
(`Datastore.Audit` + `Remote.Audit`) and use that on the PBS-NH3
remote config. Reduces blast radius if PBS-NH3 ever compromised.

### 6.4. Set retention on PBS-NH3

Shorter than primary (it's DR, not archival):

Web UI: Datastore → backups-mirror → Prune & GC → Edit:

| Field | Value |
|---|---|
| keep-last | 7 |
| keep-daily | 7 |
| keep-weekly | 4 |

### Phase 6 done-state

- Nightly pull from PBS-ANA → PBS-NH3 runs cleanly
- PBS-NH3 holds last 7 days of backups (shorter retention, DR focus)

## Phase 7 — Burn-in

**Target state:** ~1 week of clean operation before retiring legacy targets.

Daily checks:

```bash
# PBS-ANA datastore growth is sensible
ssh root@pbs-ana 'df -h /mnt/pbs-datastore'

# PBS-NH3 sync job succeeded overnight
ssh root@pbs-nh3 'journalctl -u proxmox-backup-proxy --since yesterday | grep -i sync'

# Every hypervisor's last backup is ≤24h old
# (Run refresh-proxmox-info.sh and inspect the backup-coverage section)
scripts/refresh-proxmox-info.sh all
```

Watch for:

- Failed backup jobs (email alerts if enabled)
- Sync job duration growing unexpectedly (bandwidth issue)
- Datastore approaching capacity (bump retention or storage)
- VMs that don't dirty-bitmap incremental (missing QEMU Guest Agent,
  shut-down VMs always do full dumps)

## Phase 8 — Retire legacy vzdump targets

**Target state:** no more old-style local-dump backups running; just PBS.

Only after Phase 7 passes:

- On each hypervisor: remove old vzdump jobs that targeted local dump
  dirs.
- Optionally delete old VMA files from local dump storage to reclaim
  space (before deleting, verify PBS has the same vintage).
- Update `servers/*-pve/proxmox-details.txt` by re-running
  `scripts/refresh-proxmox-info.sh all` — the `backup coverage` section
  should now show every guest covered by the PBS-ANA job.

## Phase 9 — Update this workspace

- Update `STATUS.md` coverage table to show PBS deployment state.
- Update memory `project_backup_pipeline_gaps.md` — PBS closes several
  of the open gaps.
- Update `README.md` backup pipeline section to describe the new
  topology.
- Add an entry for PBS-ANA and PBS-NH3 in `servers/` (with README +
  ssh-target). They're first-class infrastructure now.
- Homepage Infra-ANA card pointing at PBS-ANA Web UI; Infra-NH3 for
  PBS-NH3.

## Rollback posture

At any phase, you can stop and leave the fleet in its current state:

- Phase 1 incomplete: no impact, PBS-ANA just sits idle.
- Phase 2+ incomplete: affected hypervisors still have their old
  vzdump jobs running. Until Phase 8, old targets are kept as
  redundant. Rolling back one hypervisor means deleting its
  PBS storage entry and re-enabling its old job.
- Phase 6+ incomplete: PBS-ANA still provides primary backup coverage.
  DR mirror just isn't built yet.