From ad2b0e97aab5c38e9964c3b6053666b1c2c2857f Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 27 Apr 2026 20:54:08 -0700 Subject: [PATCH] docs/runbooks/nh3-prune-ritual: write up the quarterly NH3 prune ceremony MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New runbook captures the three-phase process: Phase 1 — Drop --append-only via DSM Container Manager web UI Phase 2 — sudo resticprofile forget --prune --verbose on each of nh3-docker, nh3-dev, irv-ml1 (interactive sudo per host) Phase 3 — Restore --append-only via DSM Why each phase looks the way it does, what to expect (largely no-op runs for the first 6 months while no snapshots have aged out of the keep window), how to verify each phase non-destructively (curl 401 on the rest-server root proves the container's up + serving), what to do if Phase 2 fails with `repository is configured as append-only` (skipped Phase 1 / DSM didn't apply), and the path to future automation (find docker bin path on DSM, NOPASSWD-lock syncuser to the specific recreate command). Includes a "last run history" table seeded with today's first post-pipeline run (no-op, irv-ml1 only had 3 snapshots due to the 04-25→27 CUDA stall). Cross-referenced from docs/README.md (runbook tree), docs/ orientation.md (where-to-look table), and STATUS.md item 9 (which now points at the runbook + records the next-round date 2026-07-27). --- STATUS.md | 6 + docs/README.md | 1 + docs/orientation.md | 1 + docs/runbooks/nh3-prune-ritual.md | 216 ++++++++++++++++++++++++++++++ 4 files changed, 224 insertions(+) create mode 100644 docs/runbooks/nh3-prune-ritual.md diff --git a/STATUS.md b/STATUS.md index a3b993e..f2354dd 100644 --- a/STATUS.md +++ b/STATUS.md @@ -273,6 +273,12 @@ Once recovered, update this section with what fixed it and remove the syncuser aren't cleanly scriptable from this workstation. Run with `scripts/restic-prune.sh ana|nh3|all`, optionally `--dry-run`. + **First NH3 ritual run: 2026-04-27** — full process documented in + [`docs/runbooks/nh3-prune-ritual.md`](docs/runbooks/nh3-prune-ritual.md) + (Phase 1 DSM toggle off → Phase 2 forget+prune on 3 clients → + Phase 3 DSM toggle back on). Today's run was a no-op everywhere + (every snapshot still in keep window). Next round: ~2026-07-27. + 10. ~~**Retire `offen/docker-volume-backup` sidecars**~~ — **done 2026-04-23**. Removed from paperless-ngx and pgadmin composes on esh-docker-vm (only hosts in the fleet that had them). 16 GB of diff --git a/docs/README.md b/docs/README.md index 0a480bd..ffe25a9 100644 --- a/docs/README.md +++ b/docs/README.md @@ -12,6 +12,7 @@ docs/ ├── orientation.md # start here — fleet overview + where-to-look guide ├── runbooks/ # ops runbooks (recovery, deployment phases) │ ├── disaster-recovery.md +│ ├── nh3-prune-ritual.md │ └── pbs-deployment.md └── pfi/ # PFI-specific reference (services, models, VMs) ├── docker-stack.md diff --git a/docs/orientation.md b/docs/orientation.md index 670f139..66dd5cf 100644 --- a/docs/orientation.md +++ b/docs/orientation.md @@ -189,6 +189,7 @@ These caught us once; don't let them catch you twice. | What's the backup strategy for host X? | `configs/restic//README.md` + `profiles.yaml` | | How do I recover from host X going down? | `docs/runbooks/disaster-recovery.md` | | How was PBS deployed? | `docs/runbooks/pbs-deployment.md` | +| How do I run the quarterly NH3 restic prune? | `docs/runbooks/nh3-prune-ritual.md` (or just `scripts/restic-prune.sh nh3`) | | What's currently open / in-flight? | `STATUS.md` | | What do I need to know that isn't in current code? | `MEMORY.md` + the `.md` files it links | | Why did we do X? | Check memory files + `STATUS.md` session milestones at the bottom | diff --git a/docs/runbooks/nh3-prune-ritual.md b/docs/runbooks/nh3-prune-ritual.md new file mode 100644 index 0000000..27e4c52 --- /dev/null +++ b/docs/runbooks/nh3-prune-ritual.md @@ -0,0 +1,216 @@ +# NH3 restic prune ritual + +The quarterly maintenance window where we drop `--append-only` from +`rest-server-nh3`, run `forget --prune` on every NH3-side restic +client to reclaim space from aged-out snapshots, and put the safety +guard back. **Schedule: every ~90 days**, or sooner if NH3 NAS shows +disk-pressure on `/volume1/pbs`. + +The ANA side is fully automated by `scripts/restic-prune.sh ana`. The +NH3 side has to be a half-manual ceremony because: + +* `rest-server-nh3` runs in **DSM Container Manager** on the Synology + NAS. The DSM CLI doesn't expose the docker socket on a path our + tooling expects. +* `sudo` on `syncuser` (the dedicated automation account) isn't + NOPASSWD-configured. Toggling the container env from this + workstation would require a sudo password over SSH, which our + tooling doesn't pipe. + +So the toggle parts are DSM web-UI work; the prune commands run from +your terminal with interactive sudo. Total wall time when nothing +unusual happens: **5–10 minutes**. + +`scripts/restic-prune.sh nh3` prints these same instructions +(without the running narrative) — keep that script as the source of +truth for the exact commands. This doc is the explanation + lessons +learned. + +--- + +## Why "ritual" + +`rest-server-nh3` boots with `--append-only` by default, which makes +the repo immune to ransomware that compromises a backup client: an +attacker holding nh3-docker's restic password can `backup` (add +snapshots) but can't `forget`/`prune`/`delete` any existing data. +Pruning is a privileged operation that requires the safety lifted +briefly, then restored. The "ritual" framing keeps the steps in the +right order — drop the guard, do the dangerous work, restore the +guard — so we don't accidentally leave the repo wide open. + +If something goes wrong mid-prune (network blip, client crash), the +worst case is that `--append-only` is still off when you walk away. +The repo is still passworded; an attacker would need both restic +creds AND network access to harm it. Phase 3 is mandatory but not +emergency-mandatory. + +--- + +## Phase 1 — Drop `--append-only` via DSM + +In the **DSM web UI** (Synology): + +1. Container Manager → **Container** → `rest-server` +2. **Edit** → **Environment** tab +3. Find the `OPTIONS` env var. It currently reads: + + ``` + --private-repos --append-only --prometheus + ``` + +4. Change to (drop the `--append-only`): + + ``` + --private-repos --prometheus + ``` + +5. **Apply**. DSM recreates the container with the new env. + +### Verification + +The container should come back up within ~5 seconds. From the +workstation: + +```bash +curl -sf -o /dev/null -w "root: %{http_code}\n" http://10.100.50.50:8000/ +# expect: root: 401 (container up, demanding auth — healthy) +``` + +A non-401 (000 = no response, 5xx = container crashed) means DSM +didn't recreate cleanly; check the container's logs in DSM and try +**Apply** again before moving on. + +--- + +## Phase 2 — `forget --prune` on every NH3 client + +Three clients write to `rest-server-nh3`: + +| host | what it backs up | +|---|---| +| `nh3-docker` | `/etc`, `/opt/docker`, `/root`, `/var/lib/docker/volumes` | +| `nh3-dev` | `/etc`, `/home/lkraven`, `/root` (workstation) | +| `irv-ml1` | `/etc`, `/home/lkraven`, `/opt/docker`, `/root`, `/var/lib/docker/volumes`, `/worktank/comfyui/...`, `/worktank/cosyvoice/voices` | + +Run from your TTY (each is interactive sudo): + +```bash +ssh -t nh3-docker "sudo resticprofile -c /etc/restic/profiles.yaml forget --prune --verbose" +ssh -t nh3-dev "sudo resticprofile -c /etc/restic/profiles.yaml forget --prune --verbose" +ssh -t irv-ml1 "sudo resticprofile -c /etc/restic/profiles.yaml forget --prune --verbose" +``` + +Order doesn't matter; you can also run them in parallel terminals if +you want it done faster. + +### What to expect + +* **`forget`** runs the policy from each profile (typically: keep + 7 daily, 4 weekly, 12 monthly, 3 yearly). Output lists every + snapshot that's being kept and why. +* **`prune`** then walks the repo and removes data referenced only by + the (now-forgotten) snapshots. **If `forget` removed nothing, + `prune` is silent** — nothing to free, nothing to print. +* Total per-client wall time: <1s if nothing to do, single-digit + minutes if there's a real prune to perform. + +### Likely no-op rounds + +For the first ~6 months of a fresh repo, nothing ages out — every +snapshot is still inside the keep window. The output will list all +your snapshots under "keep N snapshots:" and print "finished +'forget'" with no removals + no prune output. **This is correct**; +the ritual is a guard-restore exercise, not a data-deletion +guarantee. + +Real prune work starts when: + +* Daily snapshots older than 7 days exist (typically 2nd week of repo + life). +* Weekly snapshots older than 4 weeks (typically 2nd month). +* Monthly snapshots older than 12 months (typically 2nd year). + +### If you hit `repository is configured as append-only` + +You skipped Phase 1 or DSM didn't apply the env change. Re-do +Phase 1, verify with the curl probe, then retry. + +### If `nh3-dev` prompts for SSH host key authenticity + +That's a workstation that was rebuilt or never connected from this +particular SSH client. Type `yes` to accept the new key, then +proceed. The known-hosts entry will then be cached. + +--- + +## Phase 3 — Restore `--append-only` via DSM + +Mirror image of Phase 1: + +1. Container Manager → **Container** → `rest-server` +2. **Edit** → **Environment** +3. Set `OPTIONS` back to: + + ``` + --private-repos --append-only --prometheus + ``` + +4. **Apply**. + +### Verification + +```bash +curl -sf -o /dev/null -w "root: %{http_code}\n" http://10.100.50.50:8000/ +# expect: root: 401 +``` + +Optionally, prove `--append-only` is back by attempting a no-op +forget from any client — it should refuse with `repository is +configured as append-only`. **This is destructive-adjacent** (forget +attempts a write the server rejects), so it's safe but disruptive +to the client's restic history if logging is verbose. Generally +trust the DSM UI showing the env change applied. + +--- + +## When to schedule the next round + +Set a calendar event for **+90 days** from the last successful run. +The ANA-side automation runs the same retention policy on its own +schedule, so the two sides stay roughly in sync. + +If `df -h` on `nh3-nas` shows `/volume1/pbs` over 80%, run the +ritual sooner rather than waiting for the calendar. + +Last run history (append a line each time): + +| date | duration | notes | +|---|---|---| +| 2026-04-27 | <5 min | First ritual run since pipeline went live; no-op everywhere (every snapshot still in keep window). irv-ml1 had only 3 snapshots due to the 04-25→27 CUDA stall. | + +--- + +## Future automation hooks + +Once the DSM workflow gets tedious to repeat, the manual phases can +be replaced with `ssh -t nh3-nas` + `sudo docker exec` against the +DSM-managed container. Two prerequisites: + +1. **Find the docker binary path on DSM**. DSM Container Manager + doesn't ship `docker` on the default `$PATH` for syncuser. Probe + from an interactive shell first: + + ``` + ssh -t nh3-nas "sudo find / -name docker -executable -type f 2>/dev/null" + ``` + +2. **Set NOPASSWD sudo on syncuser** for the specific docker + recreate command. Edit `/etc/sudoers.d/syncuser-docker` carefully + — the rule should be path-locked and command-locked so a + compromised syncuser can't escalate beyond container restarts. + +When both are in place, replace the Phase 1 + Phase 3 manual blocks +in `scripts/restic-prune.sh` with shell-driven `docker container +update` (or container recreate) calls, and the whole ritual becomes +a single `scripts/restic-prune.sh nh3` invocation.