docs/runbooks/nh3-prune-ritual: write up the quarterly NH3 prune ceremony

New runbook captures the three-phase process:

  Phase 1 — Drop --append-only via DSM Container Manager web UI
  Phase 2 — sudo resticprofile forget --prune --verbose on each of
            nh3-docker, nh3-dev, irv-ml1 (interactive sudo per host)
  Phase 3 — Restore --append-only via DSM

Why each phase looks the way it does, what to expect (largely no-op
runs for the first 6 months while no snapshots have aged out of the
keep window), how to verify each phase non-destructively (curl 401
on the rest-server root proves the container's up + serving), what
to do if Phase 2 fails with `repository is configured as append-only`
(skipped Phase 1 / DSM didn't apply), and the path to future
automation (find docker bin path on DSM, NOPASSWD-lock syncuser to
the specific recreate command).

Includes a "last run history" table seeded with today's first
post-pipeline run (no-op, irv-ml1 only had 3 snapshots due to the
04-25→27 CUDA stall).

Cross-referenced from docs/README.md (runbook tree), docs/
orientation.md (where-to-look table), and STATUS.md item 9 (which
now points at the runbook + records the next-round date 2026-07-27).
This commit is contained in:
vh
2026-04-27 20:54:08 -07:00
parent ec1f5e5c8f
commit ad2b0e97aa
4 changed files with 224 additions and 0 deletions
+6
View File
@@ -273,6 +273,12 @@ Once recovered, update this section with what fixed it and remove the
syncuser aren't cleanly scriptable from this workstation. Run with
`scripts/restic-prune.sh ana|nh3|all`, optionally `--dry-run`.
**First NH3 ritual run: 2026-04-27** — full process documented in
[`docs/runbooks/nh3-prune-ritual.md`](docs/runbooks/nh3-prune-ritual.md)
(Phase 1 DSM toggle off → Phase 2 forget+prune on 3 clients →
Phase 3 DSM toggle back on). Today's run was a no-op everywhere
(every snapshot still in keep window). Next round: ~2026-07-27.
10. ~~**Retire `offen/docker-volume-backup` sidecars**~~ — **done
2026-04-23**. Removed from paperless-ngx and pgadmin composes on
esh-docker-vm (only hosts in the fleet that had them). 16 GB of