backup pipeline: configs, runbooks, NH3 Synology rest-server, cross-site rsync
Bundles the post-2026-04-21 work that built out the two-layer backup architecture (PBS for VM images + restic for file/DB), plus the cross- site mirror and the disaster-recovery runbook. - configs/restic/esh-docker-vm/profiles.yaml: drop the obsolete *_offen_backup_data exclude (offen sidecars retired fleet-wide 2026-04-23; restic now covers the equivalent scope directly). - configs/restic/esh-vm-db/: new profile for the dedicated DB VM (10.0.50.60), with pre-backup pg_dumpall + mongodump hooks. - configs/rsync/: ana-nas → nh3-nas (04:00 daily, runs as lkraven) and nh3-nas → ana-nas (05:00 daily, runs as root because DSM rest-server-nh3 writes mode-400 files only root can read). - docs/runbooks/pbs-deployment.md: 9-phase PBS rollout runbook, refined during the 2026-04-22 deployment with per-hypervisor namespaces, NFSv3 + ZFS-case-insensitivity workaround, and the Synology syno_acl flatten step. - docs/runbooks/disaster-recovery.md: blast-radius runbook ordered Tier 0 → 5 (ana-nas → hypervisors → Docker hosts → VMs → specialty); references incident memory + recovery-step playbooks per consumer.
This commit is contained in:
+195
-18
@@ -197,20 +197,77 @@ both have VM 100 for instance). Without namespaces, their backups
|
||||
collide under the same `/vm/100/` path in the datastore. Create a
|
||||
namespace per hypervisor up front:
|
||||
|
||||
`proxmox-backup-manager` doesn't manage namespaces — they're created
|
||||
through the API/web UI or the `proxmox-backup-client` tool.
|
||||
|
||||
**Easiest: web UI.** Datastore → `backups` → Content → top of pane
|
||||
there's a namespace selector with an **Add NS** button. Add one per
|
||||
hypervisor.
|
||||
|
||||
**Scripted via `proxmox-backup-client`** (run on PBS-ANA):
|
||||
|
||||
```bash
|
||||
for h in pfi-pve nh3-pve esh-pve esh-pve-nas sfsrv-ana; do
|
||||
proxmox-backup-manager namespace create backups $h
|
||||
export PBS_REPOSITORY='root@pam@localhost:backups'
|
||||
export PBS_PASSWORD='<root-pam-password>'
|
||||
for ns in pfi-pve nh3-pve esh-pve esh-pve-nas sfsrv-ana; do
|
||||
proxmox-backup-client namespace create "$ns"
|
||||
done
|
||||
proxmox-backup-manager namespace list backups
|
||||
proxmox-backup-client namespace list
|
||||
```
|
||||
|
||||
(If the CLI errors — subcommand names shift between PBS versions —
|
||||
use the web UI: Datastore → backups → Content → **Add NS**. Works
|
||||
reliably regardless of version.)
|
||||
**Scripted via API** (when ssh access is more convenient than shell
|
||||
on PBS-ANA):
|
||||
|
||||
```bash
|
||||
TOKEN='<fleet-vzdump-secret>'
|
||||
for ns in pfi-pve nh3-pve esh-pve esh-pve-nas sfsrv-ana; do
|
||||
curl -sk \
|
||||
-H "Authorization: PBSAPIToken=root@pam!fleet-vzdump:$TOKEN" \
|
||||
-X POST \
|
||||
https://10.250.50.90:8007/api2/json/admin/datastore/backups/namespace \
|
||||
-d "{\"ns\":\"$ns\"}"
|
||||
done
|
||||
```
|
||||
|
||||
Each PVE client later (Phase 2.1, 3.1, 4) sets its Namespace field to
|
||||
its own hostname when configuring the PBS storage.
|
||||
|
||||
### 1.4c. Create a verify job
|
||||
|
||||
New snapshots land unverified — PBS treats "backup completed" and
|
||||
"backup verified" as separate states. A verify job hash-checks chunk
|
||||
data (not just the manifest), flips snapshots to verified, and catches
|
||||
bitrot on aging data.
|
||||
|
||||
Web UI: **Datastore → backups → Verify Jobs → Add**:
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Schedule | `sat 23:00` |
|
||||
| Ignore verified snapshots | ✓ |
|
||||
| Re-verify after (days) | `30` |
|
||||
| Max depth | blank (unlimited) |
|
||||
| Namespace | blank (root — recurses into all namespaces) |
|
||||
| Comment | `fleet verify — new + 30d re-check` |
|
||||
|
||||
Rationale for these defaults:
|
||||
|
||||
- **`sat 23:00`** — sits between the daily 03:00 backup window and the
|
||||
Sunday 06:00 GC run, so verify never fights GC for I/O, and every
|
||||
week's new backups get verified before pruning decisions happen.
|
||||
- **`ignore-verified: true`** + **`outdated-after: 30d`** — efficient
|
||||
steady state. First run after a backup night does the new snapshots
|
||||
only; a 30-day rolling re-verify catches silent chunk corruption.
|
||||
- **unlimited depth, root namespace** — one job covers all 5
|
||||
hypervisor namespaces. Split into per-namespace jobs only if you
|
||||
want per-hypervisor visibility into verify failures (not necessary
|
||||
for a fleet this size).
|
||||
|
||||
Verify runs are I/O-heavy on the datastore — on the NFS-backed
|
||||
PBS-ANA, expect a full-datastore verify to take hours once the
|
||||
datastore grows. The `ignore-verified` flag keeps incremental verify
|
||||
cheap; only the 30-day-aged portion is re-read each run.
|
||||
|
||||
### 1.5. Create an API token for hypervisors to use
|
||||
|
||||
PBS web UI: **Configuration → Access Control → API Token → Add**:
|
||||
@@ -405,32 +462,152 @@ Same pattern as Phase 1.1:
|
||||
|
||||
Same as Phase 1.2.
|
||||
|
||||
### 5.3. Local datastore
|
||||
### 5.3. Datastore backing — Synology NFS (chosen 2026-04-22)
|
||||
|
||||
PBS-NH3 uses local storage rather than NFS (keeps the two PBSes on
|
||||
different failure domains). Two options:
|
||||
PBS-NH3 mounts a Synology NFS share rather than using a local virtual
|
||||
disk. Simpler storage admin; tradeoff is that PBS-NH3 now shares its
|
||||
failure domain with the other NH3 backup paths (nh3-docker restic,
|
||||
nh3-dev restic, Backrest repos). Acceptable for DR purposes because
|
||||
PBS-ANA remains primary.
|
||||
|
||||
**Option A:** attach a second virtual disk to the VM (e.g. 2 TB on
|
||||
nh3-pve's `local-lvm` or whatever storage is available), format ext4,
|
||||
mount at `/mnt/pbs-datastore`.
|
||||
**Synology-side export setup (DSM):**
|
||||
|
||||
**Option B:** mount Synology Btrfs share via NFS or CIFS. Simpler
|
||||
storage admin but couples PBS-NH3 to the Synology's availability.
|
||||
1. Create a dedicated share, e.g. `pbs-nh3`, on the target volume.
|
||||
2. **Control Panel → Shared Folder → [share] → Edit → NFS Permissions → Add/Edit:**
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Hostname/IP | PBS-NH3 VM IP |
|
||||
| Privilege | Read/Write |
|
||||
| Squash | **No mapping** (Synology's label for `no_root_squash`) |
|
||||
| Security | sys |
|
||||
| Enable asynchronous | on |
|
||||
| Allow connections from non-privileged ports | on |
|
||||
|
||||
Recommended: **Option A**. Keeps it self-contained.
|
||||
3. Verify no **Advanced Permissions** ACL denies `root` write — those
|
||||
override NFS perms and cause silent write failures.
|
||||
|
||||
**Client-side mount (on PBS-NH3 VM):**
|
||||
|
||||
Use **NFSv3**, not NFSv4 — see the gotcha below. Synology's "Advanced
|
||||
Permissions" layer an NFSv4 ACL on top of POSIX mode that is invisible
|
||||
to `ls` but denies writes to unprivileged users (including the
|
||||
`backup` uid-34 that PBS runs as), even when the directory mode is
|
||||
777. Root bypasses this via `no_root_squash`, which is why a root
|
||||
`touch` succeeds but the datastore init fails.
|
||||
|
||||
```bash
|
||||
# Inside the PBS-NH3 VM, after attaching a disk /dev/sdb:
|
||||
mkfs.ext4 /dev/sdb
|
||||
mkdir -p /mnt/pbs-datastore
|
||||
echo '/dev/sdb /mnt/pbs-datastore ext4 defaults 0 2' >> /etc/fstab
|
||||
|
||||
cat >> /etc/fstab <<'EOF'
|
||||
10.100.50.50:/volume1/pbs-nh3 /mnt/pbs-datastore nfs defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=3 0 0
|
||||
EOF
|
||||
|
||||
mount -a
|
||||
df -h /mnt/pbs-datastore
|
||||
|
||||
# Two-layer smoke test: root AND the backup user that PBS runs as.
|
||||
# The backup-user check is the one that actually matters — if root
|
||||
# works but backup doesn't, you're hitting the ACL override.
|
||||
touch /mnt/pbs-datastore/root-write-test && rm /mnt/pbs-datastore/root-write-test
|
||||
sudo -u backup touch /mnt/pbs-datastore/backup-user-test && rm /mnt/pbs-datastore/backup-user-test
|
||||
```
|
||||
|
||||
Ensure NFSv3 is reachable on the Synology side: **Control Panel →
|
||||
File Services → NFS → Advanced** — confirm "Minimum NFS Protocol" is
|
||||
3 (not 4.0+). Default is usually 3; only an issue if someone
|
||||
hardened it previously.
|
||||
|
||||
**Fallback if NFSv3 still fails the backup-user check:** the deny is
|
||||
a Synology-local syno_acl (`ls -la` on the Synology shows POSIX mode
|
||||
`d---------+` with a `+` for extended ACL). DSM "Enable Advanced
|
||||
Permissions" unchecked does NOT reset this once the archive flag
|
||||
`has_ACL,is_support_ACL` is set on the share.
|
||||
|
||||
Diagnose from the Synology shell:
|
||||
|
||||
```
|
||||
sudo synoacltool -get /volume1/<share>
|
||||
sudo ls -la /volume1/<share>/
|
||||
```
|
||||
|
||||
A `+` after the permission string + a `group:administrators:allow:...`
|
||||
ACL entry + POSIX `d---------` is the smoking gun: only members of
|
||||
the `administrators` group have access, which is why root (with
|
||||
`no_root_squash`) writes but uid-34 (backup) doesn't.
|
||||
|
||||
**Fix (confirmed working 2026-04-22):** keep **Squash: `No mapping`**
|
||||
(i.e. `no_root_squash`) AND flatten the share to Linux/POSIX mode
|
||||
with `chmod 777`. This drops the syno_acl entirely — verifiable by
|
||||
`synoacltool -get` returning "It's Linux mode" and `ls -la`
|
||||
showing `drwxrwxrwx` with NO trailing `+`.
|
||||
|
||||
On the Synology shell:
|
||||
|
||||
```bash
|
||||
sudo chmod 777 /volume1/<share>
|
||||
|
||||
# Verify pure POSIX, no ACL
|
||||
sudo synoacltool -get /volume1/<share> # should say "It's Linux mode"
|
||||
ls -la /volume1/<share>/ # should show drwxrwxrwx (no '+')
|
||||
```
|
||||
|
||||
Pure POSIX 777 is a cleaner long-term config than the ACL-grant
|
||||
approach — fewer permission-translation layers between NFSv3 and
|
||||
Btrfs, and no chance of ACL inheritance surprises on PBS-created
|
||||
subdirectories.
|
||||
|
||||
**Why `all_squash` alone doesn't work:** it lets backup-user writes
|
||||
through (because the ACL-granted admin gets the mapped uid), but
|
||||
breaks PBS's `chown()` during init. Squashed-admin doesn't have
|
||||
CAP_CHOWN on the Synology side → EPERM. Only real root (via
|
||||
`no_root_squash`) can chown to uid 34.
|
||||
|
||||
**Why squash alone fails:** even with `all_squash + anonuid=1024`
|
||||
letting backup-user writes succeed (because admin is in
|
||||
`administrators` ACL), PBS's datastore-init calls `chown` on
|
||||
newly-created paths. Squashed-admin doesn't have CAP_CHOWN on the
|
||||
Synology side → EPERM. Only a real root (via `no_root_squash`) can
|
||||
chown to uid 34.
|
||||
|
||||
**Smoke-test all three paths after the fix:**
|
||||
|
||||
```bash
|
||||
sudo touch /mnt/pbs-datastore/t && rm /mnt/pbs-datastore/t && echo ROOT_OK
|
||||
sudo -u backup touch /mnt/pbs-datastore/t && rm /mnt/pbs-datastore/t && echo BACKUP_OK
|
||||
touch /mnt/pbs-datastore/t && chown 34:34 /mnt/pbs-datastore/t && rm /mnt/pbs-datastore/t && echo CHOWN_OK
|
||||
```
|
||||
|
||||
All three must pass before PBS will init cleanly.
|
||||
|
||||
**Historical note:** both PBS instances ended up on NFSv3 for
|
||||
unrelated reasons — the ANA side because of ZFS case-insensitivity,
|
||||
the NH3 side because of Synology ACL override. NFSv3 is the safer
|
||||
default for PBS-on-NFS regardless of backend.
|
||||
|
||||
### 5.4. Create datastore
|
||||
|
||||
Web UI: Datastore → Add, name `backups-mirror`, path `/mnt/pbs-datastore`.
|
||||
|
||||
### 5.5. Create a verify job on the mirror
|
||||
|
||||
Same pattern as Phase 1.4c on PBS-ANA — verification doesn't replicate
|
||||
across PBS instances, so the mirror needs its own job to catch bitrot
|
||||
on the local datastore disk.
|
||||
|
||||
Web UI: **Datastore → backups-mirror → Verify Jobs → Add**:
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Schedule | `sun 12:00` |
|
||||
| Ignore verified snapshots | ✓ |
|
||||
| Re-verify after (days) | `30` |
|
||||
| Max depth | blank (unlimited) |
|
||||
| Namespace | blank |
|
||||
| Comment | `mirror verify — catches DR-side bitrot` |
|
||||
|
||||
Schedule sits after the 06:00 sync job finishes, so newly-synced
|
||||
snapshots get verified same day.
|
||||
|
||||
### Phase 5 done-state
|
||||
|
||||
- PBS-NH3 reachable, datastore ready
|
||||
|
||||
Reference in New Issue
Block a user