vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

backup pipeline: configs, runbooks, NH3 Synology rest-server, cross-site rsync

Browse Source 
Bundles the post-2026-04-21 work that built out the two-layer backup
architecture (PBS for VM images + restic for file/DB), plus the cross-
site mirror and the disaster-recovery runbook.

- configs/restic/esh-docker-vm/profiles.yaml: drop the obsolete
  *_offen_backup_data exclude (offen sidecars retired fleet-wide
  2026-04-23; restic now covers the equivalent scope directly).
- configs/restic/esh-vm-db/: new profile for the dedicated DB VM
  (10.0.50.60), with pre-backup pg_dumpall + mongodump hooks.
- configs/rsync/: ana-nas → nh3-nas (04:00 daily, runs as lkraven)
  and nh3-nas → ana-nas (05:00 daily, runs as root because DSM
  rest-server-nh3 writes mode-400 files only root can read).
- docs/runbooks/pbs-deployment.md: 9-phase PBS rollout runbook,
  refined during the 2026-04-22 deployment with per-hypervisor
  namespaces, NFSv3 + ZFS-case-insensitivity workaround, and the
  Synology syno_acl flatten step.
- docs/runbooks/disaster-recovery.md: blast-radius runbook ordered
  Tier 0 → 5 (ana-nas → hypervisors → Docker hosts → VMs → specialty);
  references incident memory + recovery-step playbooks per consumer.
This commit is contained in:
vh
2026-04-24 21:56:22 -07:00
parent 4971e5ad41
commit 574c72daa5
12 changed files with 1114 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
configs/restic/esh-docker-vm/profiles.yaml
+2 -2
View File
@@ -52,8 +52,8 @@ default:
# Docker internals # Docker internals
- /var/lib/docker/volumes/backingFsBlockDev - /var/lib/docker/volumes/backingFsBlockDev
- /var/lib/docker/volumes/metadata.db - /var/lib/docker/volumes/metadata.db
# Offen sidecar's buffer / tmp (if any left behind) # Offen `*_offen_backup_data` exclude removed 2026-04-23 — offen
- /var/lib/docker/volumes/*_offen_backup_data # sidecars retired fleet-wide; no more offen-scratch volumes exist.
# Ephemeral / regenerable # Ephemeral / regenerable
- /opt/docker/compose/*/logs - /opt/docker/compose/*/logs
- "**/*.log" - "**/*.log"
configs/restic/esh-vm-db/README.md
+164
View File
@@ -0,0 +1,164 @@
# restic / esh-vm-db
**Two-database host** at the ESH site (PostgreSQL 15 + MongoDB). Covered
at the VM-image layer by PBS-ANA via esh-pve (or whichever ESH
hypervisor owns this VM — confirm on next inventory pass). This restic
profile adds DB-level granularity via pre-backup dumps.
## What's backed up
| Path | Purpose |
|---|---|
| `/etc` | Host config — systemd, ssh, chrony, apt, pg_hba.conf, mongod.conf |
| `/root` | Root's ad-hoc scripts, shell history, ssh keys |
| `/home` | User homes (lkraven + any DB-admin locals) |
| `/var/lib/restic/stage` | **pg_dumpall.sql.gz** + **mongodump/** produced by pre-backup.sh |
## What's **not** backed up (by design)
- **`/var/lib/postgresql`** — raw PGDATA. Live-capture risk;
`pg_dumpall` in pre-backup covers it consistently.
- **`/var/lib/mongodb`** — raw mongo dbPath. Same reasoning;
`mongodump` covers it.
- NFS mount `/mnt/backup` (from esh-nas — not ours to mirror).
## Pre-backup hook
`pre-backup.sh` runs as root before restic. It:
1. Checks `pg_isready` on :5432 — if OK, runs `pg_dumpall` piped
through gzip to `$STAGE/pg_dumpall.sql.gz`
2. Checks mongo ping via `mongosh` — if OK, runs `mongodump` into
`$STAGE/mongodump/`
Both dumps are atomic (write to `.tmp`, then rename). If either DB is
unreachable, the script logs a WARN and continues — a failed DB dump
doesn't abort the whole restic run, and restic falls back to whatever
stage content is left over from the prior successful dump.
## Deploy (one-time)
### 1. Create rest-server-ana htpasswd entry
**Do NOT use `sudo` for .htpasswd writes on ana-docker.** The file is
NFS-mounted from ana-nas and owned by uid 1000 (the rest-server user,
which equals lkraven). Sudo-root on the client gets squashed to
nobody on the NFS server and can't read/write the file. lkraven
writes it natively, using the docker group for the bcrypt helper.
```bash
# Pick password in password manager first
HTPW='<new-pw-saved-to-pw-manager>'
ssh -t ana-docker "docker run --rm httpd:2.4-alpine htpasswd -nbB esh-vm-db '$HTPW' | \
tee /tmp/htline.txt > /dev/null && \
sed -i '/^esh-vm-db:/d' /mnt/backup/restic/repo/ana/.htpasswd && \
cat /tmp/htline.txt >> /mnt/backup/restic/repo/ana/.htpasswd && \
rm /tmp/htline.txt && \
grep ^esh-vm-db: /mnt/backup/restic/repo/ana/.htpasswd && \
docker restart rest-server"
unset HTPW
```
### 2. Install secrets on esh-vm-db
```bash
ssh -t esh-vm-db 'sudo install -d -o root -g root -m 0700 /etc/restic /var/lib/restic /var/lib/restic/stage'
# restic.env — URL-encode the password if it has special chars
ssh -t esh-vm-db "sudo bash -c '
read -sp \"htpasswd pw for rest-server-ana: \" HTPW; echo
cat > /etc/restic/restic.env <<EOF
RESTIC_REPOSITORY=rest:http://esh-vm-db:\$HTPW@10.250.50.70:8000/esh-vm-db/
EOF
chmod 600 /etc/restic/restic.env
'"
# Repo passphrase (prints once — save to password manager)
ssh -t esh-vm-db 'sudo bash -c "
openssl rand -base64 48 | tr -d \"\\n\" > /etc/restic/password
chmod 600 /etc/restic/password
echo === SAVE THIS TO PASSWORD MANAGER NOW ===
cat /etc/restic/password
echo
"'
```
### 3. Initialize the repo
```bash
ssh -t esh-vm-db 'sudo bash -c "
set -a; . /etc/restic/restic.env; set +a
RESTIC_PASSWORD_FILE=/etc/restic/password restic init
"'
```
### 4. Install prerequisites (restic, resticprofile, mongosh client)
```bash
ssh -t esh-vm-db 'which restic || sudo apt-get install -y restic; \
which mongosh || echo "NOTE: mongosh not found; pre-backup mongo ping will fail safely — install via MongoDB APT repo if needed"; \
curl -sfL https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh | sudo sh -s -- -b /usr/local/bin; \
/usr/local/bin/resticprofile --version'
```
### 5. Deploy profile + hook
```bash
scp configs/restic/esh-vm-db/profiles.yaml esh-vm-db:/tmp/
scp configs/restic/esh-vm-db/pre-backup.sh esh-vm-db:/tmp/
ssh -t esh-vm-db 'sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml && \
sudo install -o root -g root -m 0755 /tmp/pre-backup.sh /etc/restic/pre-backup.sh && \
rm /tmp/profiles.yaml /tmp/pre-backup.sh'
```
### 6. Schedule + verify
```bash
ssh -t esh-vm-db 'sudo resticprofile --config /etc/restic/profiles.yaml schedule --all && \
systemctl list-timers "resticprofile*" --no-pager'
# First manual run
ssh -t esh-vm-db 'sudo resticprofile --config /etc/restic/profiles.yaml backup --verbose'
```
Expect first run to land ~50-200 MB (mostly the mongodump directory + pg_dumpall).
Cross-check from Backrest UI on ana-docker.
## Restore
### Full host config
```bash
ssh -t esh-vm-db 'sudo resticprofile --config /etc/restic/profiles.yaml restore latest --target /tmp/restore --path /etc'
```
### Just the PG dump
```bash
ssh -t esh-vm-db 'sudo resticprofile --config /etc/restic/profiles.yaml restore latest --target /tmp/restore --path /var/lib/restic/stage/pg_dumpall.sql.gz'
# Then: gunzip + psql < pg_dumpall.sql
```
### Just a mongo DB
```bash
ssh -t esh-vm-db 'sudo resticprofile --config /etc/restic/profiles.yaml restore latest --target /tmp/restore --path /var/lib/restic/stage/mongodump'
# Then: mongorestore /tmp/restore/var/lib/restic/stage/mongodump/
```
## Gotchas
- **mongosh must be installed** or the mongo pre-backup step silently
skips (logged as WARN). Install from the MongoDB APT repo if not
already present — the stock Debian `mongodb-clients` package is
out of date and doesn't include `mongosh`.
- **Mongo authentication** — if mongod ever gets auth enabled (it's
currently open to 0.0.0.0 with no auth, which is its own concern),
`mongodump` will need `--username/--password` flags. Reference in
pre-backup.sh when that change happens.
- **pg_hba.conf** — `pg_dumpall` requires local postgres superuser
access. Currently works via `sudo -u postgres` + peer auth on the
local socket. If `pg_hba.conf` ever changes peer → md5 for local,
the hook needs a `~postgres/.pgpass` entry.
configs/restic/esh-vm-db/pre-backup.sh
+59
View File
@@ -0,0 +1,59 @@
#!/bin/bash
# pre-backup.sh — esh-vm-db.
# Runs as root from resticprofile's `run-before`.
#
# Produces consistent DB dumps in /var/lib/restic/stage/. Two DBs here:
# - Postgres 15 (port 5432, local) — pg_dumpall all databases
# - MongoDB (port 27017, local) — mongodump all databases
#
# Peer-data DBs (paperless-ng on Postgres) are the primary consumers;
# raw volume capture isn't in the restic source list so these dumps
# are the ONLY way restic sees DB data.
#
# Errors in individual blocks log a WARN; whole script doesn't abort.
set -euo pipefail
STAGE=/var/lib/restic/stage
install -d -o root -g root -m 0700 "$STAGE"
log() { printf '%s pre-backup(esh-vm-db): %s\n' "$(date -Is)" "$*"; }
warn() { printf '%s pre-backup(esh-vm-db): WARN: %s\n' "$(date -Is)" "$*" >&2; }
# ---- Postgres ----------------------------------------------------------
PG_DUMP="$STAGE/pg_dumpall.sql.gz"
if sudo -u postgres pg_isready -h localhost -p 5432 > /dev/null 2>&1; then
log "pg_dumpall starting → $PG_DUMP"
if sudo -u postgres pg_dumpall -h localhost -p 5432 | gzip > "$PG_DUMP.tmp"; then
mv "$PG_DUMP.tmp" "$PG_DUMP"
log "pg_dumpall done ($(du -h "$PG_DUMP" | cut -f1))"
else
warn "pg_dumpall failed (exit $?); keeping previous dump if any"
rm -f "$PG_DUMP.tmp"
fi
else
warn "postgres not ready on :5432 — skipping pg_dumpall"
fi
# ---- MongoDB -----------------------------------------------------------
MONGO_DIR="$STAGE/mongodump"
if mongosh --quiet --eval 'db.adminCommand({ping: 1}).ok' | grep -q '^1$'; then
log "mongodump starting → $MONGO_DIR"
rm -rf "$MONGO_DIR.tmp"
if mongodump --out "$MONGO_DIR.tmp" > /dev/null 2>&1; then
rm -rf "$MONGO_DIR"
mv "$MONGO_DIR.tmp" "$MONGO_DIR"
log "mongodump done ($(du -sh "$MONGO_DIR" | cut -f1))"
else
warn "mongodump failed (exit $?); keeping previous dump if any"
rm -rf "$MONGO_DIR.tmp"
fi
else
warn "mongo not reachable via mongosh — skipping mongodump"
fi
# ---- Retention on stage dir --------------------------------------------
# restic dedupes identical dumps at the chunk level, so we can safely keep
# overwriting the same files. No explicit rotation needed here.
log "pre-backup complete"
configs/restic/esh-vm-db/profiles.yaml
+89
View File
@@ -0,0 +1,89 @@
# resticprofile config for esh-vm-db.
#
# Two-database host (Postgres 15 on :5432, MongoDB on :27017). Small
# VM on an ESH hypervisor; backed up at VM-image level by PBS-ANA via
# its hypervisor. This restic profile captures:
#
# 1. Host config (/etc, /root) — fast config recovery without
# waiting for VM-image restore
# 2. Per-DB dumps via pre-backup hook — pg_dumpall + mongodump,
# written to /var/lib/restic/stage/ and included in the restic
# snapshot. Gives us DB-level restore granularity alongside the
# VM-image restore from PBS.
#
# What's DELIBERATELY NOT in source:
# - /var/lib/postgresql — raw live PGDATA. Inconsistent if captured
# while postgres is running; pg_dumpall above covers this.
# - /var/lib/mongodb — same reasoning; mongodump covers it.
#
# Writes cross-site to rest-server-ana (10.250.50.70:8000). No local
# NH3-style rest-server on the ESH side; esh-docker-vm and vm-esh-nas
# also use rest-server-ana, so this follows fleet pattern.
version: "1"
global:
priority: low
ionice: true
ionice-class: 2
ionice-level: 7
min-memory: 100
default:
env-file: /etc/restic/restic.env
env:
RESTIC_PASSWORD_FILE: /etc/restic/password
initialize: false
lock: /var/lock/restic-esh-vm-db.lock
backup:
verbose: 1
run-before:
- /etc/restic/pre-backup.sh
run-after:
- date +%s > /var/lib/restic/last-success
source:
- /etc # host config
- /root # root's scripts, ssh keys, shell history
- /home # user home dirs
- /var/lib/restic/stage # pg_dumpall + mongodump outputs from pre-backup
exclude:
# Raw DB data is captured via pre-backup dumps, not volume-level
- /var/lib/postgresql
- /var/lib/mongodb
# NFS mount from esh-nas (backup target for other stacks — not ours to mirror)
- /mnt/backup
# Ephemeral / regenerable
- "**/*.log"
- "**/*.log.*"
- "**/*.pid"
- /root/.cache
- /root/.local/share/Trash
- /root/.npm
- /root/.python_history
- /home/*/.cache
- /home/*/.local/share/Trash
- /home/*/.npm
tag:
- host:esh-vm-db
- site:esh
- fleet:home-lab
schedule: "*-*-* 01:00:00"
schedule-permission: system
schedule-log: /var/log/restic-backup.log
forget:
keep-daily: 7
keep-weekly: 4
keep-monthly: 12
keep-yearly: 3
tag:
- host:esh-vm-db
# Schedule removed: forget against --append-only rest-server always
# fails (delete ops blocked). Run manually during prune ceremony.
check:
read-data-subset: 10%
schedule: "Sun *-*-* 05:00:00"
schedule-permission: system
schedule-log: /var/log/restic-check.log
configs/rsync/ana-nas-to-nh3/README.md
+96
View File
@@ -0,0 +1,96 @@
# ana-nas → nh3-nas restic mirror
Nightly rsync of the Ana-side restic repo to the NH3 Synology, giving
file-level backups cross-site redundancy independent of the PBS layer.
## What + where
| | |
|---|---|
| **Source** | `ana-nas:/mnt/backup/restic/repo/ana/` (ZFS `NASPool/backupStore`, written by rest-server-ana on ana-docker via NFS). |
| **Target** | `nh3-nas:/volume1/Backup/restic-ana-mirror/` (Btrfs `/volume1`, 16 TB free at setup time). |
| **Runs on** | `ana-nas` (Debian 12, VMID 100 on pfi-pve). |
| **Runs as** | `lkraven` (uid 1000) — owns the source data natively; no sudo needed. |
| **Auth** | Dedicated ed25519 keypair `~lkraven/.ssh/id_mirror_nh3` → `syncuser@nh3-nas`. No passphrase (for unattended systemd runs). |
| **Schedule** | Daily at 04:00, `Persistent=true`, 300s randomized delay. |
## Scheduling rationale
Window sits between:
- **01:00–03:00** — ana-side restic clients run their nightly backups
against rest-server-ana. We wait until those are clearly done before
reading the repo.
- **06:00** — PBS-ANA → PBS-NH3 sync job pulls fresh snapshots across
the WAN. Running the restic mirror earlier avoids competing for WAN
bandwidth with the (much larger) PBS sync.
## Files
| File | Install path on ana-nas |
|---|---|
| `restic-mirror-to-nh3.service` | `/etc/systemd/system/restic-mirror-to-nh3.service` |
| `restic-mirror-to-nh3.timer` | `/etc/systemd/system/restic-mirror-to-nh3.timer` |
These are the canonical copies; the on-host copies mirror them.
## Deploy / redeploy
```bash
cd configs/rsync/ana-nas-to-nh3
scp restic-mirror-to-nh3.{service,timer} ana-nas:/tmp/
ssh -t ana-nas 'sudo install -m 644 /tmp/restic-mirror-to-nh3.service /etc/systemd/system/ && \
sudo install -m 644 /tmp/restic-mirror-to-nh3.timer /etc/systemd/system/ && \
sudo systemctl daemon-reload && \
sudo systemctl enable --now restic-mirror-to-nh3.timer && \
sudo systemctl list-timers restic-mirror-to-nh3.timer'
```
## Operate
```bash
# Trigger a run manually (e.g. to smoke-test changes)
ssh -t ana-nas 'sudo systemctl start restic-mirror-to-nh3.service'
# Follow a running transfer
ssh ana-nas 'journalctl -u restic-mirror-to-nh3.service -f'
# Last run outcome
ssh ana-nas 'systemctl status restic-mirror-to-nh3.service --no-pager'
# When the timer fires next
ssh ana-nas 'systemctl list-timers restic-mirror-to-nh3.timer'
```
## Restore flow (if ana-nas loses the repo)
The mirror is just files — point a restic client directly at the
nh3-nas copy. Options:
**Option A — pull repo back to ana-nas and use rest-server-ana as
before.** rsync in reverse:
```bash
ssh -t ana-nas 'sudo rsync -a \
-e "ssh -i /home/lkraven/.ssh/id_mirror_nh3" \
syncuser@10.100.50.50:/volume1/Backup/restic-ana-mirror/ \
/mnt/backup/restic/repo/ana/'
```
**Option B — restic against the mirror directly.** Expose
`/volume1/Backup/restic-ana-mirror/` via a temporary rest-server or
NFS share, point the client at it for an emergency restore. Slow (WAN
hop) but no dataset copy required.
Target is `--append-only: false` on the mirror side — not a problem
for restore, but means a compromised mirror side *could* be tampered
with. Balanced against the operational cost of managing an
append-only mirror, we accept this for a mirror-of-a-mirror.
## What this does NOT cover
- **PBS snapshots** — those replicate via PBS-ANA → PBS-NH3 (a
separate pipeline; see `docs/runbooks/pbs-deployment.md`).
- **rest-server-nh3's own data** — nh3-docker + nh3-dev backups that
land on `/volume1/Backup/restic/<user>/`. Those are already at the
NH3 Synology; ANA-side doesn't mirror them currently. Symmetric
mirror back to ana-nas is a future-work item.
configs/rsync/ana-nas-to-nh3/restic-mirror-to-nh3.service
+46
View File
@@ -0,0 +1,46 @@
[Unit]
Description=Mirror rest-server-ana restic repo to NH3 Synology
Documentation=https://github.com/lkraven/eshpfi-management/blob/main/configs/rsync/ana-nas-to-nh3/README.md
After=network-online.target
Wants=network-online.target
ConditionPathIsDirectory=/mnt/backup/restic/repo/ana
StartLimitBurst=3
StartLimitIntervalSec=1h
[Service]
Type=oneshot
User=lkraven
Group=lkraven
Nice=10
IOSchedulingClass=idle
# Dedicated key for this job, ed25519, ana-nas → nh3-nas:syncuser.
# --append-only on rest-server-ana means source files are never
# rewritten or deleted by clients; --delete here mirrors any explicit
# prune operations (done out-of-band during the quarterly ceremony).
# tmp/ and .lock excluded to avoid mirroring in-flight transfers.
ExecStart=/usr/bin/rsync \
--archive \
--delete \
--partial \
--info=stats2 \
--timeout=300 \
--exclude=tmp/ \
--exclude=.lock \
-e "ssh -i /home/lkraven/.ssh/id_mirror_nh3 -o StrictHostKeyChecking=accept-new -o BatchMode=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o ConnectTimeout=30" \
/mnt/backup/restic/repo/ana/ \
syncuser@10.100.50.50:/volume1/Backup/restic-ana-mirror/
# ServerAlive{Interval,CountMax}=30/3 + rsync --timeout bound below
# force detection of dead WAN within ~90s; without this, a silent TCP
# drop holds the service in "activating" for hours on kernel retransmit
# backoff. TimeoutStartSec=6h caps worst-case initial sync; after
# that systemd kills the service even if rsync somehow wedges.
TimeoutStartSec=6h
# Tight retry window — a transient WAN blip shouldn't wedge the timer
# schedule, but spamming retries on a real outage is noise.
# StartLimitBurst/Interval live in [Unit] (systemd v230+); the burst
# cap prevents runaway retries even with the short RestartSec.
Restart=on-failure
RestartSec=30s
configs/rsync/ana-nas-to-nh3/restic-mirror-to-nh3.timer
+17
View File
@@ -0,0 +1,17 @@
[Unit]
Description=Daily mirror of rest-server-ana restic repo to NH3
Documentation=https://github.com/lkraven/eshpfi-management/blob/main/configs/rsync/ana-nas-to-nh3/README.md
[Timer]
# 04:00 daily — sits between ana-side restic clients (01:00 finish
# window) and the PBS-ANA → PBS-NH3 sync at 06:00, so WAN contention
# is minimized. Persistent=true catches missed runs if ana-nas was
# offline; RandomizedDelaySec smears load if this ever scales to
# multiple mirror jobs on the same host.
OnCalendar=*-*-* 04:00:00
Persistent=true
RandomizedDelaySec=300
Unit=restic-mirror-to-nh3.service
[Install]
WantedBy=timers.target
configs/rsync/nh3-nas-to-ana/README.md
+104
View File
@@ -0,0 +1,104 @@
# nh3-nas → ana-nas restic mirror (return leg)
Symmetric counterpart to `configs/rsync/ana-nas-to-nh3/`. Nightly
rsync of the NH3-side restic repo to the Ana NAS, so file-level
backups for nh3-docker and nh3-dev have cross-site redundancy.
## What + where
| | |
|---|---|
| **Source** | `nh3-nas:/volume1/Backup/restic/` (Btrfs on cachedev_0, written by rest-server-nh3 on nh3-nas). Holds per-client repos: nh3-docker/, nh3-dev/, repo/. |
| **Target** | `ana-nas:/mnt/backup/restic-nh3-mirror/` (ZFS `NASPool/backupStore`, 20 TB pool with ~20 TB free at setup). |
| **Runs on** | `nh3-nas` (Synology RS2418+, DSM 7+ with native systemd). |
| **Runs as** | **`root`** — required because rest-server-nh3's Docker container writes restic files as `admin:users mode 400`, and only root (via CAP_DAC_READ_SEARCH) can read them. Running as `syncuser` was tried first and denied. See gotchas below. |
| **Auth** | Dedicated ed25519 keypair `/root/.ssh/id_mirror_ana` → `lkraven@ana-nas`. No passphrase. |
| **Schedule** | Daily at 05:00, `Persistent=true`, 300s randomized delay. |
## Why 05:00
Slots between:
- 04:00 ana-nas → nh3-nas (the other mirror) — lets that finish before
NH3 starts pushing anything back.
- 06:00 PBS-ANA → PBS-NH3 sync — avoids WAN contention with the much
larger PBS replication.
## DSM-specific notes
- DSM 7.x ships a stripped/older systemd. Unit files in
`/etc/systemd/system/` persist across DSM updates but are NOT
guaranteed to — verify after each DSM major update.
- **DSM systemd is pre-v247** — it refuses `Restart=on-failure` on
`Type=oneshot` services, and does not recognize `StartLimitBurst`
or `StartLimitIntervalSec`. This service deliberately omits those;
a failed run just means the daily timer picks up again tomorrow.
For a mirror used as DR (not primary), that's acceptable.
- `systemctl --now` flag is also absent. Use `enable` + `start`
separately on DSM.
- `syncuser` has `administrators` group membership, so sudo works
for unit installation. `syncuser` itself runs the service (no
privilege escalation at runtime).
- rsync lives at `/usr/bin/rsync` on DSM (same as Debian). Older
`/bin/rsync` doesn't exist.
- `scp` to DSM needs `-O` (legacy protocol) — sftp subsystem is off
by default. Either enable SFTP in DSM (**Control Panel → File
Services → FTP → SFTP**) or keep using `scp -O`.
- **rest-server-nh3 writes restic files as `admin:users mode 400`**
(via DSM ContainerManager). syncuser cannot read these even with
admins-group ACL because the newer DSM doesn't ship `synoacltool`
to add per-user ACEs. The service therefore runs as root
(CAP_DAC_READ_SEARCH bypasses the 400 mode).
- **rsync `--archive` broke cross-filesystem perm/owner preservation**
from Btrfs-with-syno_acl → ZFS-on-Debian — the dest dir ended up
mode `0000` and rsync couldn't recover on subsequent runs. Switched
to `-rlt --no-perms --no-owner --no-group --chmod=D755,F644` which
copies contents verbatim and forces sensible dest perms. Restic's
repo integrity doesn't rely on Unix perms so this is a clean swap.
## Files
| File | Install path on nh3-nas |
|---|---|
| `restic-mirror-to-ana.service` | `/etc/systemd/system/restic-mirror-to-ana.service` |
| `restic-mirror-to-ana.timer` | `/etc/systemd/system/restic-mirror-to-ana.timer` |
## Prereqs (one-time setup)
1. Generate the keypair on nh3-nas as syncuser:
```bash
ssh nh3-nas "test -f ~/.ssh/id_mirror_ana || ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_mirror_ana -C 'nh3-nas→ana-nas restic mirror'"
```
2. Install the pubkey on ana-nas:
```bash
ssh nh3-nas 'cat ~/.ssh/id_mirror_ana.pub' | \
ssh ana-nas 'cat >> ~/.ssh/authorized_keys'
```
3. Create the destination dir on ana-nas:
```bash
ssh ana-nas 'sudo mkdir -p /mnt/backup/restic-nh3-mirror && \
sudo chown lkraven:lkraven /mnt/backup/restic-nh3-mirror'
```
4. Smoke-test from nh3-nas as syncuser:
```bash
ssh nh3-nas 'ssh -i ~/.ssh/id_mirror_ana -o StrictHostKeyChecking=accept-new \
lkraven@10.250.50.50 "hostname && ls -ld /mnt/backup/restic-nh3-mirror"'
```
## Deploy
```bash
cd configs/rsync/nh3-nas-to-ana
scp restic-mirror-to-ana.{service,timer} nh3-nas:/tmp/
ssh -t nh3-nas 'sudo install -m 644 /tmp/restic-mirror-to-ana.service /etc/systemd/system/ && \
sudo install -m 644 /tmp/restic-mirror-to-ana.timer /etc/systemd/system/ && \
sudo systemctl daemon-reload && \
sudo systemctl enable --now restic-mirror-to-ana.timer && \
sudo systemctl start restic-mirror-to-ana.service && \
sudo systemctl list-timers restic-mirror-to-ana.timer'
```
## Operate
Mirrors the ana-nas side — see that README. Replace
`restic-mirror-to-nh3` with `restic-mirror-to-ana` and `ana-nas` with
`nh3-nas` in the commands there.
configs/rsync/nh3-nas-to-ana/restic-mirror-to-ana.service
+52
View File
@@ -0,0 +1,52 @@
[Unit]
Description=Mirror rest-server-nh3 restic repo to ANA NAS
Documentation=https://github.com/lkraven/eshpfi-management/blob/main/configs/rsync/nh3-nas-to-ana/README.md
After=network-online.target
Wants=network-online.target
ConditionPathIsDirectory=/volume1/Backup/restic
[Service]
Type=oneshot
User=root
Group=root
Nice=10
IOSchedulingClass=idle
# Runs as root because rest-server-nh3's Docker container writes the
# restic repo files as admin:users mode 400 — only root bypasses that
# via CAP_DAC_READ_SEARCH. Running as syncuser (even with admins
# group membership) was denied by POSIX mode. Earlier attempt with
# synoacltool ACL grants failed because this DSM version ships
# without the tool.
#
# Dedicated key for this job, ed25519, nh3-nas:root → ana-nas:lkraven.
# Does NOT include /volume1/Backup/restic-ana-mirror (that's data
# ana-nas just sent us; mirroring it back would be a dedupe-less loop).
# Source is the per-host tree under /volume1/Backup/restic/ only:
# nh3-docker/, nh3-dev/, repo/.
# tmp/ and .lock excluded to avoid mirroring in-flight transfers.
# SSH keepalive + rsync --timeout ensure a dead WAN is detected in
# ~90s instead of hanging on kernel TCP retransmit backoff.
#
# NO Restart=on-failure here: DSM's systemd is pre-v247 and refuses
# Restart= on Type=oneshot. If a run fails, the daily timer picks
# up again tomorrow — a single missed mirror is acceptable for DR.
ExecStart=/usr/bin/rsync \
--recursive \
--links \
--times \
--no-perms \
--no-owner \
--no-group \
--chmod=D755,F644 \
--delete \
--partial \
--info=stats2 \
--timeout=300 \
--exclude=tmp/ \
--exclude=.lock \
-e "ssh -i /root/.ssh/id_mirror_ana -o StrictHostKeyChecking=accept-new -o BatchMode=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o ConnectTimeout=30" \
/volume1/Backup/restic/ \
lkraven@10.250.50.50:/mnt/backup/restic-nh3-mirror/
TimeoutStartSec=6h
configs/rsync/nh3-nas-to-ana/restic-mirror-to-ana.timer
+14
View File
@@ -0,0 +1,14 @@
[Unit]
Description=Daily mirror of rest-server-nh3 restic repo to ANA
Documentation=https://github.com/lkraven/eshpfi-management/blob/main/configs/rsync/nh3-nas-to-ana/README.md
[Timer]
# 05:00 daily — runs after ana→nh3 at 04:00, before PBS sync at
# 06:00. Same persistence + jitter pattern.
OnCalendar=*-*-* 05:00:00
Persistent=true
RandomizedDelaySec=300
Unit=restic-mirror-to-ana.service
[Install]
WantedBy=timers.target
docs/runbooks/disaster-recovery.md
+276
View File
@@ -0,0 +1,276 @@
# Fleet disaster-recovery runbook
What breaks when a given host/service goes down, and how to recover.
Ordered by blast-radius severity — read top-down.
**First action for ANY multi-service outage:** check ana-nas
reachability. Many fleet-wide "things are broken" events trace back
to ana-nas's NFS exports going stale. See [ana-nas SPOF memory](../../../.claude/projects/-home-lkraven-development-eshpfi-management/memory/incident_ana_nas_spof.md)
for details, or just `ping 10.250.50.50` first.
## Governing principle: backups must not risk production
Any backup strategy that could take down the host it's backing up is
worse than the risk it mitigates. Specifically applies to:
- **NFS-serving hosts (CT 109 / ana-nas):** scheduled vzdump of the
container itself is prohibited. The rootfs is trivially rebuildable;
the data lives on ospool bind-mounts captured via ZFS snapshot at
the host level. If rootfs config is worth preserving, capture via
`rsync /etc /root` to a known location — no snapshot, no freeze.
- **DB servers on NFS storage (VM 105 / pfi-postgres):** prefer
`pg_dump` + restic over full vzdump. fs-freeze on NFS-backed
PGDATA is a failure mode waiting to happen.
- **Any host with heavy I/O concurrency during the backup window:**
schedule its own backup outside the fleet window OR use `mode=stop`
(planned brief downtime beats random-freeze risk).
Incident reference: 2026-04-23 — CT 109, 112, 113 all went offline on
pfi-pve during morning backup window. Backup suspected as trigger
(full root cause TBD). See `memory/incident_ana_nas_spof.md`.
---
## Tier 0 — Storage fabric (catastrophic blast radius)
### ana-nas (CT 109 on pfi-pve, 10.250.50.50)
**Blast radius:**
- ~~pfi-postgres (VM 105) — PGDATA on `/mnt/db`~~ — **migrated to local disk 2026-04-23**. vaultwarden, gitea, paperless-ng, zammad no longer cascade on ana-nas outage. Left in history for the recovery pre-migration.
- ana-docker rest-server-ana — repo data on `/mnt/backup` → all ana-side restic clients fail (ana-docker, ana-ml2, esh-docker-vm, vm-esh-nas)
- PBS-ANA datastore — NFS-backed on `/mnt/backup/pbs-ana` → fleet vzdumps fail, PBS-NH3 sync fails
- ana-docker NFS mounts for `/mnt/docker`, `/mnt/compose`, `/mnt/pve-VMStorage` if used → various stack misbehavior
**Recovery** (see full procedure in `memory/incident_ana_nas_spof.md`):
1. `pct start 109` on pfi-pve if stopped; `pct console 109` to see boot if stuck
2. Once responsive, remount NFS on each consumer and restart services:
- pfi-postgres: `umount -lf /mnt/db; mount /mnt/db; systemctl restart postgresql`
- ana-docker: `sudo mount -a; sudo docker restart rest-server` — usually enough. `mount -a` re-attempts all fstab entries and bypasses the "failed" state that `mnt-backup.mount` gets stuck in (fstab uses bare `defaults` without auto-retry). If rest-server still errors with `/data/.htpasswd: permission denied`, you've got a **ghost file on the local mount point** — see incident memory for the clean-up procedure (stop rest-server, unmount NFS, rm the ghost, remount, restart).
- PBS-ANA: `umount -lf /mnt/pbs-datastore; mount /mnt/pbs-datastore; systemctl restart proxmox-backup proxmox-backup-proxy`
3. Re-run failed PBS vzdump job for affected VMs (Datacenter → Backup → Run Now)
4. Restic clients recover automatically on next 01:00 schedule
**Prevention:**
- Re-schedule ana-nas's own vzdump OUTSIDE the 03:00 fleet window (concurrent NFS load during fleet backup + self-backup is the suspected 2026-04-23 crash root cause)
- Consider moving pfi-postgres PGDATA to local VM disk (reduces SPOF)
---
### nh3-nas (Synology RS2418+, 10.100.50.50)
**Blast radius:**
- nh3-docker, nh3-dev restic clients — writes to `rest-server-nh3` fail
- PBS-NH3 datastore — mirror sync can't write new chunks
- ana-nas → nh3-nas rsync (04:00 daily) — fails
**Recovery:**
1. Power-cycle via DSM web UI or physical button if unresponsive
2. DSM boots 3-5 minutes; NFS exports auto-start
3. Verify PBS-NH3 datastore recovered:
```
ssh pbs-nh3 'ls /mnt/pbs-datastore/.chunks | head; systemctl restart proxmox-backup proxmox-backup-proxy'
```
4. ana-nas-side rsync retries at next 04:00; no manual action unless you want to trigger now
**Notable:** nh3-nas outage is LESS severe than ana-nas because:
- No production DB depends on it
- It's a mirror/DR tier, not primary
- Ana-side backups keep running independently
---
## Tier 1 — Hypervisors (regional blast radius)
### pfi-pve (Proxmox VE, 10.250.250.31)
**Blast radius — everything hosted on it:**
- ana-docker (VM), ana-nas (CT 109), pfi-postgres (VM 105), pfi-ana-webhost, ana-filebot, pfi-pteradactyl, pfi-tacticalrmm, ana-wg, PBS-ANA (VM), + others
Essentially **all Anaheim primary services** go offline. Because ana-nas lives here too, Tier-0 cascade applies simultaneously.
**Recovery:**
1. Is the Proxmox host itself up? iDRAC/IPMI console for hardware diagnosis.
2. If the host is up but VMs/CTs aren't starting: `systemctl status pve-cluster qemu-server pve-container` on the host.
3. Start VMs/CTs in dependency order:
- **First**: CT 109 (ana-nas) — everything else on the host that uses NFS depends on it
- **Second**: PBS-ANA VM (resume backup target)
- **Third**: VM 105 (pfi-postgres) — many apps wait on this
- **Fourth**: ana-docker, then other VMs
4. After all VMs/CTs up, re-apply the Tier-0 ana-nas recovery steps for any consumers with stale NFS mounts.
**Prevention:** PBS-ANA → PBS-NH3 nightly sync means VM image backups are recoverable at NH3 if pfi-pve is unrecoverable; can restore key VMs to nh3-pve and cut over (cold DR).
---
### nh3-pve (10.100.250.60)
**Blast radius:**
- nh3-docker (VM) — NH3 site's Docker stacks
- PBS-NH3 (VM) — DR mirror target
- Other NH3 VMs
**Recovery:** same pattern as pfi-pve. PBS-NH3 being down breaks DR replication but doesn't affect primary backups (ANA side is authoritative).
---
### esh-pve + esh-pve-nas (10.0.250.35 + 10.0.50.55)
**Blast radius:**
- All ESH home-lab workloads: esh-docker-vm, vm-esh-nas, media services, home automation stacks
**Recovery:** restore VMs from PBS-ANA (cross-site pull). Not production-critical; no SLA.
---
### sfsrv-ana (Dell R630, 10.250.250.115, iDRAC 10.250.250.110)
**Blast radius:** SureFire tenant workloads.
- PBS-ANA namespace `sfsrv-pve` has daily snapshots
- Coordinate with tenant before any recovery action per hosting agreement
**Recovery:** restore from PBS-ANA sfsrv-pve namespace. Tenant may have app-level recovery procedures that take precedence.
---
## Tier 2 — Docker hosts (per-stack blast radius)
### ana-docker (VM 10.250.50.70)
**Blast radius — services hosted on it:**
- rest-server-ana (fleet restic target; depends on ana-nas NFS)
- Backrest UI, Beszel hub, Dozzle hub, Traefik (ANA), CrowdSec, Gitea, Vaultwarden, Seafile, Synapse, OpenWebUI, SearXNG, IT Tools, others
**Recovery:**
1. VM restart on pfi-pve: `qm stop <vmid>; qm start <vmid>`
2. After boot, stacks auto-start via `docker compose up -d` (compose files at `/opt/docker/compose/*/`)
3. Check each stack with `docker ps` / dockge
4. If `/mnt/backup` (NFS) is stale after pfi-pve reboot, force remount
### nh3-docker (VM 10.100.50.40)
**Blast radius:**
- rest-server-nh3 (if migrated there — currently on nh3-nas Synology directly)
- NH3-only stacks
**Recovery:** VM restart on nh3-pve. Cross-site Beszel/Dozzle agents on ana-docker keep reporting independently.
### esh-docker-vm + vm-esh-nas
**Blast radius:** ESH home-lab Docker workloads. Paperless, HA, CWA, pgadmin (esh-docker-vm); dockge, filezilla, agents (vm-esh-nas).
**Recovery:** VM restart on respective ESH hypervisors. Not production-critical.
---
## Tier 3 — Workload VMs/LXCs
### pfi-postgres (VM 105 on pfi-pve)
**Blast radius:** vaultwarden, gitea, paperless-ng databases.
**Recovery:** usually it's actually an ana-nas issue (see Tier 0). If postgres itself is the problem:
1. Check logs: `sudo tail -200 /var/lib/postgresql/*/main/log/postgresql-*.log`
2. If clean: `sudo systemctl restart postgresql`
3. If WAL corruption: restore from PBS vzdump (most recent snapshot)
### ana-wg (CT 113 on pfi-pve) — WireGuard VPN
**Blast radius:** remote-access VPN down; remote admin sessions drop but on-prem ops continue.
**Recovery:** `pct start 113` on pfi-pve. Restart wg-quick service if needed.
### pfi-pteradactyl (VM 107 on pfi-pve) — game panel
**Blast radius:** hosted game servers offline.
**Recovery:** VM restart. Not operationally critical.
### pfi-tacticalrmm (VM 111 on pfi-pve) — RMM tooling
**Blast radius:** RMM agent dashboards; agents continue running on endpoints but can't phone home.
**Recovery:** VM restart; TacticalRMM services auto-start.
### ana-filebot (CT 112 on pfi-pve) — file automation
**Blast radius:** scheduled file operations (rename, organize); low-criticality.
**Recovery:** `pct start 112`.
### pfi-ana-webhost (VM on pfi-pve) — web workload
**Blast radius:** hosted website(s) offline.
**Recovery:** VM restart; web server auto-start.
---
## Tier 4 — Specialty workloads
### ana-ml2 (bare metal Supermicro, 10.250.50.54, BMC 10.250.250.50)
**Blast radius:** AI inference services (llama-swap, vllm-qwen3). Consumer-facing chat/embedding endpoints fail.
**Recovery:**
1. Check OS via SSH. If unresponsive, BMC console at <https://10.250.250.50>.
2. If hardware issue: BMC logs, power cycle via IPMI, check GPU health (`nvidia-smi`).
3. Docker stacks auto-start via compose `restart: unless-stopped`.
**Notable:** no Proxmox hypervisor involved — recovery is pure bare-metal. File-level restic runs from inside the host. No vzdump coverage (it's not a VM). Backups via restic to rest-server-ana.
### SureFire hosts (sfsrv-ana, sf-ana-container, sf-r630)
**Blast radius:** tenant workloads (scoped to SureFire client).
**Recovery:** coordinate with tenant. PFI responsibility is hardware + OS layer; application recovery may need tenant input. See `servers/sfsrv-ana/README.md` for hosting agreement scope.
---
## Tier 5 — Networking
### FortiGate 60F (ESH gateway, 10.0.250.1)
**Blast radius:** ESH site WAN + inter-site VPN to ANA/NH3.
**Recovery:** physical console access; restore config from FortiManager if needed.
### UniFi controllers (ESH 10.0.0.1, PFI 10.100.0.1)
**Blast radius:** UniFi AP/switch management (existing config persists on devices; only changes need the controller).
**Recovery:** device power cycle; controller reboot. Not critical for ongoing operations.
---
## Cross-cutting: what to check FIRST for ambiguous outages
When symptoms are vague ("lots of things are down"), run this triage in order:
```bash
# 1. Is the Anaheim NAS alive?
ping -c 2 10.250.50.50
# 2. Are the hypervisors alive?
for h in 10.250.250.31 10.100.250.60 10.0.250.35 10.0.50.55 10.250.250.115; do
ping -c 1 -W 2 $h >/dev/null && echo "$h OK" || echo "$h DOWN"
done
# 3. WAN reachability between sites?
# (from nh3-dev to ana-side IP; from ana-side to nh3-side IP)
ping -c 2 10.250.50.70 # ana-docker from NH3
```
The first `DOWN` in the hypervisor list narrows blast radius to that site / that hypervisor's guests.
---
## What this runbook does NOT cover
- Application-level restores from PBS / restic snapshots — that's per-service and lives in respective stack READMEs
- Full site-failover (e.g., "move PFI fleet to NH3 entirely") — requires Phase-9-style DR plan, not built
- Data-level integrity recovery (DB corruption, ZFS pool damage) — beyond this runbook's scope
**Maintenance note:** update this file when:
- New host gets registered (add to applicable tier)
- New SPOF discovered (add blast-radius note)
- A recovery procedure changes in practice
docs/runbooks/pbs-deployment.md
+195 -18
View File
@@ -197,20 +197,77 @@ both have VM 100 for instance). Without namespaces, their backups
collide under the same `/vm/100/` path in the datastore. Create a collide under the same `/vm/100/` path in the datastore. Create a
namespace per hypervisor up front: namespace per hypervisor up front:
`proxmox-backup-manager` doesn't manage namespaces — they're created
through the API/web UI or the `proxmox-backup-client` tool.
**Easiest: web UI.** Datastore → `backups` → Content → top of pane
there's a namespace selector with an **Add NS** button. Add one per
hypervisor.
**Scripted via `proxmox-backup-client`** (run on PBS-ANA):
```bash ```bash
for h in pfi-pve nh3-pve esh-pve esh-pve-nas sfsrv-ana; do export PBS_REPOSITORY='root@pam@localhost:backups'
proxmox-backup-manager namespace create backups $h export PBS_PASSWORD='<root-pam-password>'
for ns in pfi-pve nh3-pve esh-pve esh-pve-nas sfsrv-ana; do
proxmox-backup-client namespace create "$ns"
done done
proxmox-backup-manager namespace list backups proxmox-backup-client namespace list
``` ```
(If the CLI errors — subcommand names shift between PBS versions — **Scripted via API** (when ssh access is more convenient than shell
use the web UI: Datastore → backups → Content → **Add NS**. Works on PBS-ANA):
reliably regardless of version.)
```bash
TOKEN='<fleet-vzdump-secret>'
for ns in pfi-pve nh3-pve esh-pve esh-pve-nas sfsrv-ana; do
curl -sk \
-H "Authorization: PBSAPIToken=root@pam!fleet-vzdump:$TOKEN" \
-X POST \
https://10.250.50.90:8007/api2/json/admin/datastore/backups/namespace \
-d "{\"ns\":\"$ns\"}"
done
```
Each PVE client later (Phase 2.1, 3.1, 4) sets its Namespace field to Each PVE client later (Phase 2.1, 3.1, 4) sets its Namespace field to
its own hostname when configuring the PBS storage. its own hostname when configuring the PBS storage.
### 1.4c. Create a verify job
New snapshots land unverified — PBS treats "backup completed" and
"backup verified" as separate states. A verify job hash-checks chunk
data (not just the manifest), flips snapshots to verified, and catches
bitrot on aging data.
Web UI: **Datastore → backups → Verify Jobs → Add**:
| Field | Value |
|---|---|
| Schedule | `sat 23:00` |
| Ignore verified snapshots | ✓ |
| Re-verify after (days) | `30` |
| Max depth | blank (unlimited) |
| Namespace | blank (root — recurses into all namespaces) |
| Comment | `fleet verify — new + 30d re-check` |
Rationale for these defaults:
- **`sat 23:00`** — sits between the daily 03:00 backup window and the
Sunday 06:00 GC run, so verify never fights GC for I/O, and every
week's new backups get verified before pruning decisions happen.
- **`ignore-verified: true`** + **`outdated-after: 30d`** — efficient
steady state. First run after a backup night does the new snapshots
only; a 30-day rolling re-verify catches silent chunk corruption.
- **unlimited depth, root namespace** — one job covers all 5
hypervisor namespaces. Split into per-namespace jobs only if you
want per-hypervisor visibility into verify failures (not necessary
for a fleet this size).
Verify runs are I/O-heavy on the datastore — on the NFS-backed
PBS-ANA, expect a full-datastore verify to take hours once the
datastore grows. The `ignore-verified` flag keeps incremental verify
cheap; only the 30-day-aged portion is re-read each run.
### 1.5. Create an API token for hypervisors to use ### 1.5. Create an API token for hypervisors to use
PBS web UI: **Configuration → Access Control → API Token → Add**: PBS web UI: **Configuration → Access Control → API Token → Add**:
@@ -405,32 +462,152 @@ Same pattern as Phase 1.1:
Same as Phase 1.2. Same as Phase 1.2.
### 5.3. Local datastore ### 5.3. Datastore backing — Synology NFS (chosen 2026-04-22)
PBS-NH3 uses local storage rather than NFS (keeps the two PBSes on PBS-NH3 mounts a Synology NFS share rather than using a local virtual
different failure domains). Two options: disk. Simpler storage admin; tradeoff is that PBS-NH3 now shares its
failure domain with the other NH3 backup paths (nh3-docker restic,
nh3-dev restic, Backrest repos). Acceptable for DR purposes because
PBS-ANA remains primary.
**Option A:** attach a second virtual disk to the VM (e.g. 2 TB on **Synology-side export setup (DSM):**
nh3-pve's `local-lvm` or whatever storage is available), format ext4,
mount at `/mnt/pbs-datastore`.
**Option B:** mount Synology Btrfs share via NFS or CIFS. Simpler 1. Create a dedicated share, e.g. `pbs-nh3`, on the target volume.
storage admin but couples PBS-NH3 to the Synology's availability. 2. **Control Panel → Shared Folder → [share] → Edit → NFS Permissions → Add/Edit:**
| Field | Value |
|---|---|
| Hostname/IP | PBS-NH3 VM IP |
| Privilege | Read/Write |
| Squash | **No mapping** (Synology's label for `no_root_squash`) |
| Security | sys |
| Enable asynchronous | on |
| Allow connections from non-privileged ports | on |
Recommended: **Option A**. Keeps it self-contained. 3. Verify no **Advanced Permissions** ACL denies `root` write — those
override NFS perms and cause silent write failures.
**Client-side mount (on PBS-NH3 VM):**
Use **NFSv3**, not NFSv4 — see the gotcha below. Synology's "Advanced
Permissions" layer an NFSv4 ACL on top of POSIX mode that is invisible
to `ls` but denies writes to unprivileged users (including the
`backup` uid-34 that PBS runs as), even when the directory mode is
777. Root bypasses this via `no_root_squash`, which is why a root
`touch` succeeds but the datastore init fails.
```bash ```bash
# Inside the PBS-NH3 VM, after attaching a disk /dev/sdb:
mkfs.ext4 /dev/sdb
mkdir -p /mnt/pbs-datastore mkdir -p /mnt/pbs-datastore
echo '/dev/sdb /mnt/pbs-datastore ext4 defaults 0 2' >> /etc/fstab
cat >> /etc/fstab <<'EOF'
10.100.50.50:/volume1/pbs-nh3 /mnt/pbs-datastore nfs defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=3 0 0
EOF
mount -a mount -a
df -h /mnt/pbs-datastore
# Two-layer smoke test: root AND the backup user that PBS runs as.
# The backup-user check is the one that actually matters — if root
# works but backup doesn't, you're hitting the ACL override.
touch /mnt/pbs-datastore/root-write-test && rm /mnt/pbs-datastore/root-write-test
sudo -u backup touch /mnt/pbs-datastore/backup-user-test && rm /mnt/pbs-datastore/backup-user-test
``` ```
Ensure NFSv3 is reachable on the Synology side: **Control Panel →
File Services → NFS → Advanced** — confirm "Minimum NFS Protocol" is
3 (not 4.0+). Default is usually 3; only an issue if someone
hardened it previously.
**Fallback if NFSv3 still fails the backup-user check:** the deny is
a Synology-local syno_acl (`ls -la` on the Synology shows POSIX mode
`d---------+` with a `+` for extended ACL). DSM "Enable Advanced
Permissions" unchecked does NOT reset this once the archive flag
`has_ACL,is_support_ACL` is set on the share.
Diagnose from the Synology shell:
```
sudo synoacltool -get /volume1/<share>
sudo ls -la /volume1/<share>/
```
A `+` after the permission string + a `group:administrators:allow:...`
ACL entry + POSIX `d---------` is the smoking gun: only members of
the `administrators` group have access, which is why root (with
`no_root_squash`) writes but uid-34 (backup) doesn't.
**Fix (confirmed working 2026-04-22):** keep **Squash: `No mapping`**
(i.e. `no_root_squash`) AND flatten the share to Linux/POSIX mode
with `chmod 777`. This drops the syno_acl entirely — verifiable by
`synoacltool -get` returning "It's Linux mode" and `ls -la`
showing `drwxrwxrwx` with NO trailing `+`.
On the Synology shell:
```bash
sudo chmod 777 /volume1/<share>
# Verify pure POSIX, no ACL
sudo synoacltool -get /volume1/<share> # should say "It's Linux mode"
ls -la /volume1/<share>/ # should show drwxrwxrwx (no '+')
```
Pure POSIX 777 is a cleaner long-term config than the ACL-grant
approach — fewer permission-translation layers between NFSv3 and
Btrfs, and no chance of ACL inheritance surprises on PBS-created
subdirectories.
**Why `all_squash` alone doesn't work:** it lets backup-user writes
through (because the ACL-granted admin gets the mapped uid), but
breaks PBS's `chown()` during init. Squashed-admin doesn't have
CAP_CHOWN on the Synology side → EPERM. Only real root (via
`no_root_squash`) can chown to uid 34.
**Why squash alone fails:** even with `all_squash + anonuid=1024`
letting backup-user writes succeed (because admin is in
`administrators` ACL), PBS's datastore-init calls `chown` on
newly-created paths. Squashed-admin doesn't have CAP_CHOWN on the
Synology side → EPERM. Only a real root (via `no_root_squash`) can
chown to uid 34.
**Smoke-test all three paths after the fix:**
```bash
sudo touch /mnt/pbs-datastore/t && rm /mnt/pbs-datastore/t && echo ROOT_OK
sudo -u backup touch /mnt/pbs-datastore/t && rm /mnt/pbs-datastore/t && echo BACKUP_OK
touch /mnt/pbs-datastore/t && chown 34:34 /mnt/pbs-datastore/t && rm /mnt/pbs-datastore/t && echo CHOWN_OK
```
All three must pass before PBS will init cleanly.
**Historical note:** both PBS instances ended up on NFSv3 for
unrelated reasons — the ANA side because of ZFS case-insensitivity,
the NH3 side because of Synology ACL override. NFSv3 is the safer
default for PBS-on-NFS regardless of backend.
### 5.4. Create datastore ### 5.4. Create datastore
Web UI: Datastore → Add, name `backups-mirror`, path `/mnt/pbs-datastore`. Web UI: Datastore → Add, name `backups-mirror`, path `/mnt/pbs-datastore`.
### 5.5. Create a verify job on the mirror
Same pattern as Phase 1.4c on PBS-ANA — verification doesn't replicate
across PBS instances, so the mirror needs its own job to catch bitrot
on the local datastore disk.
Web UI: **Datastore → backups-mirror → Verify Jobs → Add**:
| Field | Value |
|---|---|
| Schedule | `sun 12:00` |
| Ignore verified snapshots | ✓ |
| Re-verify after (days) | `30` |
| Max depth | blank (unlimited) |
| Namespace | blank |
| Comment | `mirror verify — catches DR-side bitrot` |
Schedule sits after the 06:00 sync job finishes, so newly-synced
snapshots get verified same day.
### Phase 5 done-state ### Phase 5 done-state
- PBS-NH3 reachable, datastore ready - PBS-NH3 reachable, datastore ready