docs/runbooks/nh3-prune-ritual: write up the quarterly NH3 prune ceremony

New runbook captures the three-phase process:

  Phase 1 — Drop --append-only via DSM Container Manager web UI
  Phase 2 — sudo resticprofile forget --prune --verbose on each of
            nh3-docker, nh3-dev, irv-ml1 (interactive sudo per host)
  Phase 3 — Restore --append-only via DSM

Why each phase looks the way it does, what to expect (largely no-op
runs for the first 6 months while no snapshots have aged out of the
keep window), how to verify each phase non-destructively (curl 401
on the rest-server root proves the container's up + serving), what
to do if Phase 2 fails with `repository is configured as append-only`
(skipped Phase 1 / DSM didn't apply), and the path to future
automation (find docker bin path on DSM, NOPASSWD-lock syncuser to
the specific recreate command).

Includes a "last run history" table seeded with today's first
post-pipeline run (no-op, irv-ml1 only had 3 snapshots due to the
04-25→27 CUDA stall).

Cross-referenced from docs/README.md (runbook tree), docs/
orientation.md (where-to-look table), and STATUS.md item 9 (which
now points at the runbook + records the next-round date 2026-07-27).

STATUS.md

@@ -273,6 +273,12 @@ Once recovered, update this section with what fixed it and remove the
    syncuser aren't cleanly scriptable from this workstation. Run with
    `scripts/restic-prune.sh ana|nh3|all`, optionally `--dry-run`.
 
+   **First NH3 ritual run: 2026-04-27** — full process documented in
+   [`docs/runbooks/nh3-prune-ritual.md`](docs/runbooks/nh3-prune-ritual.md)
+   (Phase 1 DSM toggle off → Phase 2 forget+prune on 3 clients →
+   Phase 3 DSM toggle back on). Today's run was a no-op everywhere
+   (every snapshot still in keep window). Next round: ~2026-07-27.
+
 10. ~~**Retire `offen/docker-volume-backup` sidecars**~~ — **done
     2026-04-23**. Removed from paperless-ngx and pgadmin composes on
     esh-docker-vm (only hosts in the fleet that had them). 16 GB of

docs/README.md

@@ -12,6 +12,7 @@ docs/
 ├── orientation.md            # start here — fleet overview + where-to-look guide
 ├── runbooks/                 # ops runbooks (recovery, deployment phases)
 │   ├── disaster-recovery.md
+│   ├── nh3-prune-ritual.md
 │   └── pbs-deployment.md
 └── pfi/                      # PFI-specific reference (services, models, VMs)
     ├── docker-stack.md

docs/orientation.md

@@ -189,6 +189,7 @@ These caught us once; don't let them catch you twice.
 | What's the backup strategy for host X? | `configs/restic/<host>/README.md` + `profiles.yaml` |
 | How do I recover from host X going down? | `docs/runbooks/disaster-recovery.md` |
 | How was PBS deployed? | `docs/runbooks/pbs-deployment.md` |
+| How do I run the quarterly NH3 restic prune? | `docs/runbooks/nh3-prune-ritual.md` (or just `scripts/restic-prune.sh nh3`) |
 | What's currently open / in-flight? | `STATUS.md` |
 | What do I need to know that isn't in current code? | `MEMORY.md` + the `.md` files it links |
 | Why did we do X? | Check memory files + `STATUS.md` session milestones at the bottom |

docs/runbooks/nh3-prune-ritual.md

@@ -0,0 +1,216 @@
+# NH3 restic prune ritual
+
+The quarterly maintenance window where we drop `--append-only` from
+`rest-server-nh3`, run `forget --prune` on every NH3-side restic
+client to reclaim space from aged-out snapshots, and put the safety
+guard back. **Schedule: every ~90 days**, or sooner if NH3 NAS shows
+disk-pressure on `/volume1/pbs`.
+
+The ANA side is fully automated by `scripts/restic-prune.sh ana`. The
+NH3 side has to be a half-manual ceremony because:
+
+* `rest-server-nh3` runs in **DSM Container Manager** on the Synology
+  NAS. The DSM CLI doesn't expose the docker socket on a path our
+  tooling expects.
+* `sudo` on `syncuser` (the dedicated automation account) isn't
+  NOPASSWD-configured. Toggling the container env from this
+  workstation would require a sudo password over SSH, which our
+  tooling doesn't pipe.
+
+So the toggle parts are DSM web-UI work; the prune commands run from
+your terminal with interactive sudo. Total wall time when nothing
+unusual happens: **5–10 minutes**.
+
+`scripts/restic-prune.sh nh3` prints these same instructions
+(without the running narrative) — keep that script as the source of
+truth for the exact commands. This doc is the explanation + lessons
+learned.
+
+---
+
+## Why "ritual"
+
+`rest-server-nh3` boots with `--append-only` by default, which makes
+the repo immune to ransomware that compromises a backup client: an
+attacker holding nh3-docker's restic password can `backup` (add
+snapshots) but can't `forget`/`prune`/`delete` any existing data.
+Pruning is a privileged operation that requires the safety lifted
+briefly, then restored. The "ritual" framing keeps the steps in the
+right order — drop the guard, do the dangerous work, restore the
+guard — so we don't accidentally leave the repo wide open.
+
+If something goes wrong mid-prune (network blip, client crash), the
+worst case is that `--append-only` is still off when you walk away.
+The repo is still passworded; an attacker would need both restic
+creds AND network access to harm it. Phase 3 is mandatory but not
+emergency-mandatory.
+
+---
+
+## Phase 1 — Drop `--append-only` via DSM
+
+In the **DSM web UI** (Synology):
+
+1. Container Manager → **Container** → `rest-server`
+2. **Edit** → **Environment** tab
+3. Find the `OPTIONS` env var. It currently reads:
+
+   ```
+   --private-repos --append-only --prometheus
+   ```
+
+4. Change to (drop the `--append-only`):
+
+   ```
+   --private-repos --prometheus
+   ```
+
+5. **Apply**. DSM recreates the container with the new env.
+
+### Verification
+
+The container should come back up within ~5 seconds. From the
+workstation:
+
+```bash
+curl -sf -o /dev/null -w "root: %{http_code}\n" http://10.100.50.50:8000/
+# expect: root: 401   (container up, demanding auth — healthy)
+```
+
+A non-401 (000 = no response, 5xx = container crashed) means DSM
+didn't recreate cleanly; check the container's logs in DSM and try
+**Apply** again before moving on.
+
+---
+
+## Phase 2 — `forget --prune` on every NH3 client
+
+Three clients write to `rest-server-nh3`:
+
+| host | what it backs up |
+|---|---|
+| `nh3-docker` | `/etc`, `/opt/docker`, `/root`, `/var/lib/docker/volumes` |
+| `nh3-dev` | `/etc`, `/home/lkraven`, `/root` (workstation) |
+| `irv-ml1` | `/etc`, `/home/lkraven`, `/opt/docker`, `/root`, `/var/lib/docker/volumes`, `/worktank/comfyui/...`, `/worktank/cosyvoice/voices` |
+
+Run from your TTY (each is interactive sudo):
+
+```bash
+ssh -t nh3-docker "sudo resticprofile -c /etc/restic/profiles.yaml forget --prune --verbose"
+ssh -t nh3-dev    "sudo resticprofile -c /etc/restic/profiles.yaml forget --prune --verbose"
+ssh -t irv-ml1    "sudo resticprofile -c /etc/restic/profiles.yaml forget --prune --verbose"
+```
+
+Order doesn't matter; you can also run them in parallel terminals if
+you want it done faster.
+
+### What to expect
+
+* **`forget`** runs the policy from each profile (typically: keep
+  7 daily, 4 weekly, 12 monthly, 3 yearly). Output lists every
+  snapshot that's being kept and why.
+* **`prune`** then walks the repo and removes data referenced only by
+  the (now-forgotten) snapshots. **If `forget` removed nothing,
+  `prune` is silent** — nothing to free, nothing to print.
+* Total per-client wall time: <1s if nothing to do, single-digit
+  minutes if there's a real prune to perform.
+
+### Likely no-op rounds
+
+For the first ~6 months of a fresh repo, nothing ages out — every
+snapshot is still inside the keep window. The output will list all
+your snapshots under "keep N snapshots:" and print "finished
+'forget'" with no removals + no prune output. **This is correct**;
+the ritual is a guard-restore exercise, not a data-deletion
+guarantee.
+
+Real prune work starts when:
+
+* Daily snapshots older than 7 days exist (typically 2nd week of repo
+  life).
+* Weekly snapshots older than 4 weeks (typically 2nd month).
+* Monthly snapshots older than 12 months (typically 2nd year).
+
+### If you hit `repository is configured as append-only`
+
+You skipped Phase 1 or DSM didn't apply the env change. Re-do
+Phase 1, verify with the curl probe, then retry.
+
+### If `nh3-dev` prompts for SSH host key authenticity
+
+That's a workstation that was rebuilt or never connected from this
+particular SSH client. Type `yes` to accept the new key, then
+proceed. The known-hosts entry will then be cached.
+
+---
+
+## Phase 3 — Restore `--append-only` via DSM
+
+Mirror image of Phase 1:
+
+1. Container Manager → **Container** → `rest-server`
+2. **Edit** → **Environment**
+3. Set `OPTIONS` back to:
+
+   ```
+   --private-repos --append-only --prometheus
+   ```
+
+4. **Apply**.
+
+### Verification
+
+```bash
+curl -sf -o /dev/null -w "root: %{http_code}\n" http://10.100.50.50:8000/
+# expect: root: 401
+```
+
+Optionally, prove `--append-only` is back by attempting a no-op
+forget from any client — it should refuse with `repository is
+configured as append-only`. **This is destructive-adjacent** (forget
+attempts a write the server rejects), so it's safe but disruptive
+to the client's restic history if logging is verbose. Generally
+trust the DSM UI showing the env change applied.
+
+---
+
+## When to schedule the next round
+
+Set a calendar event for **+90 days** from the last successful run.
+The ANA-side automation runs the same retention policy on its own
+schedule, so the two sides stay roughly in sync.
+
+If `df -h` on `nh3-nas` shows `/volume1/pbs` over 80%, run the
+ritual sooner rather than waiting for the calendar.
+
+Last run history (append a line each time):
+
+| date | duration | notes |
+|---|---|---|
+| 2026-04-27 | <5 min | First ritual run since pipeline went live; no-op everywhere (every snapshot still in keep window). irv-ml1 had only 3 snapshots due to the 04-25→27 CUDA stall. |
+
+---
+
+## Future automation hooks
+
+Once the DSM workflow gets tedious to repeat, the manual phases can
+be replaced with `ssh -t nh3-nas` + `sudo docker exec` against the
+DSM-managed container. Two prerequisites:
+
+1. **Find the docker binary path on DSM**. DSM Container Manager
+   doesn't ship `docker` on the default `$PATH` for syncuser. Probe
+   from an interactive shell first:
+
+   ```
+   ssh -t nh3-nas "sudo find / -name docker -executable -type f 2>/dev/null"
+   ```
+
+2. **Set NOPASSWD sudo on syncuser** for the specific docker
+   recreate command. Edit `/etc/sudoers.d/syncuser-docker` carefully
+   — the rule should be path-locked and command-locked so a
+   compromised syncuser can't escalate beyond container restarts.
+
+When both are in place, replace the Phase 1 + Phase 3 manual blocks
+in `scripts/restic-prune.sh` with shell-driven `docker container
+update` (or container recreate) calls, and the whole ritual becomes
+a single `scripts/restic-prune.sh nh3` invocation.