New runbook captures the three-phase process:
Phase 1 — Drop --append-only via DSM Container Manager web UI
Phase 2 — sudo resticprofile forget --prune --verbose on each of
nh3-docker, nh3-dev, irv-ml1 (interactive sudo per host)
Phase 3 — Restore --append-only via DSM
Why each phase looks the way it does, what to expect (largely no-op
runs for the first 6 months while no snapshots have aged out of the
keep window), how to verify each phase non-destructively (curl 401
on the rest-server root proves the container's up + serving), what
to do if Phase 2 fails with `repository is configured as append-only`
(skipped Phase 1 / DSM didn't apply), and the path to future
automation (find docker bin path on DSM, NOPASSWD-lock syncuser to
the specific recreate command).
Includes a "last run history" table seeded with today's first
post-pipeline run (no-op, irv-ml1 only had 3 snapshots due to the
04-25→27 CUDA stall).
Cross-referenced from docs/README.md (runbook tree), docs/
orientation.md (where-to-look table), and STATUS.md item 9 (which
now points at the runbook + records the next-round date 2026-07-27).
Bundles the post-2026-04-21 work that built out the two-layer backup
architecture (PBS for VM images + restic for file/DB), plus the cross-
site mirror and the disaster-recovery runbook.
- configs/restic/esh-docker-vm/profiles.yaml: drop the obsolete
*_offen_backup_data exclude (offen sidecars retired fleet-wide
2026-04-23; restic now covers the equivalent scope directly).
- configs/restic/esh-vm-db/: new profile for the dedicated DB VM
(10.0.50.60), with pre-backup pg_dumpall + mongodump hooks.
- configs/rsync/: ana-nas → nh3-nas (04:00 daily, runs as lkraven)
and nh3-nas → ana-nas (05:00 daily, runs as root because DSM
rest-server-nh3 writes mode-400 files only root can read).
- docs/runbooks/pbs-deployment.md: 9-phase PBS rollout runbook,
refined during the 2026-04-22 deployment with per-hypervisor
namespaces, NFSv3 + ZFS-case-insensitivity workaround, and the
Synology syno_acl flatten step.
- docs/runbooks/disaster-recovery.md: blast-radius runbook ordered
Tier 0 → 5 (ana-nas → hypervisors → Docker hosts → VMs → specialty);
references incident memory + recovery-step playbooks per consumer.
VMIDs aren't globally unique across PVE hosts (esh-pve and sfsrv-ana
both have VM 100, etc.). Without namespaces, per-hypervisor backups
collide under the same /vm/<vmid>/ path in the shared datastore.
Changes:
- New Phase 1.4b: create one namespace per hypervisor up front
(pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana).
- Phase 2.1 storage-entry template now lists Namespace as a required
field, set to the hypervisor's own name.
- Critical-note explaining the collision risk so future deployers
don't skip this step.
ACL grants remain at the datastore level; they apply across all child
namespaces so the existing fleet-vzdump token continues to work. Sync
job (Phase 6) preserves namespace tree to PBS-NH3 automatically.
Discovered during Phase 1.3 (mount NAS datastore). The Debian NAS's
/mnt/backup is on ZFS with casesensitivity=insensitive; NFSv4 writes
fail with EACCES even for root with no_root_squash. Known-bad
combination at the ZFS-on-Linux + NFSv4 layer.
Workaround: mount the NFS share with vers=3. PBS's chunk-based
access pattern works fine over NFSv3.
Added a section 0.5 to Phase 0 documenting the issue + both fixes
(quick: use NFSv3; cleaner: create case-sensitive child dataset).
Future deployments against this NAS won't rediscover the same wall.
Current NFS exports on the Debian NAS (10.250.50.50) use root_squash,
which blocks PBS from writing its datastore metadata (chunks, locks,
GC state — all root-owned operations). Rest-server-ana worked around
this by running its container as UID 1000, but PBS's service model
doesn't accommodate that pattern cleanly.
Solution baked into Phase 0: create a dedicated NFS export for the
PBS-ANA datastore subtree, scoped to only the PBS-ANA VM's IP, with
no_root_squash. Bounded exposure (single client), kept in a separate
.exports file so Cockpit's File Sharing module doesn't clobber it.
Flag world-scoped export on /mnt/pve-VMStorage as a non-blocking
hygiene item for a later Cockpit pass.
End-to-end runbook for standing up Proxmox Backup Server across the
fleet. Path A architecture: single primary at ANA, one-way sync to NH3
for disaster recovery. All 5 hypervisors (pfi-pve, nh3-pve, esh-pve,
esh-pve-nas, sfsrv-ana) migrate from local-dump vzdump to PBS-ANA.
Key decisions captured in the runbook:
- PBS in a Debian VM (not LXC) for clean capability model.
- PBS-ANA on pfi-pve, datastore via NFS from 10.250.50.50 —
separates backup data from hypervisor boot disk.
- PBS-NH3 on nh3-pve with local storage (independent failure
domain from ANA).
- Dedicated fleet-vzdump API token; read-only sync token for
PBS-NH3's pull job.
- sfsrv-ana specifically goes from zero backup coverage to full
vzdump coverage in Phase 3.
9 phases, each self-contained with a done-state and rollback
posture. User can stop between phases without leaving the fleet in a
bad state.
STATUS.md: added item 6b tracking this deployment. Original item 6
(cross-site rsync) now scoped to restic-only since PBS handles the
VM-image cross-site redundancy directly.
vh