docs/runbooks/nh3-prune-ritual: write up the quarterly NH3 prune ceremony
New runbook captures the three-phase process:
Phase 1 — Drop --append-only via DSM Container Manager web UI
Phase 2 — sudo resticprofile forget --prune --verbose on each of
nh3-docker, nh3-dev, irv-ml1 (interactive sudo per host)
Phase 3 — Restore --append-only via DSM
Why each phase looks the way it does, what to expect (largely no-op
runs for the first 6 months while no snapshots have aged out of the
keep window), how to verify each phase non-destructively (curl 401
on the rest-server root proves the container's up + serving), what
to do if Phase 2 fails with `repository is configured as append-only`
(skipped Phase 1 / DSM didn't apply), and the path to future
automation (find docker bin path on DSM, NOPASSWD-lock syncuser to
the specific recreate command).
Includes a "last run history" table seeded with today's first
post-pipeline run (no-op, irv-ml1 only had 3 snapshots due to the
04-25→27 CUDA stall).
Cross-referenced from docs/README.md (runbook tree), docs/
orientation.md (where-to-look table), and STATUS.md item 9 (which
now points at the runbook + records the next-round date 2026-07-27).
This commit is contained in:
@@ -189,6 +189,7 @@ These caught us once; don't let them catch you twice.
|
||||
| What's the backup strategy for host X? | `configs/restic/<host>/README.md` + `profiles.yaml` |
|
||||
| How do I recover from host X going down? | `docs/runbooks/disaster-recovery.md` |
|
||||
| How was PBS deployed? | `docs/runbooks/pbs-deployment.md` |
|
||||
| How do I run the quarterly NH3 restic prune? | `docs/runbooks/nh3-prune-ritual.md` (or just `scripts/restic-prune.sh nh3`) |
|
||||
| What's currently open / in-flight? | `STATUS.md` |
|
||||
| What do I need to know that isn't in current code? | `MEMORY.md` + the `.md` files it links |
|
||||
| Why did we do X? | Check memory files + `STATUS.md` session milestones at the bottom |
|
||||
|
||||
Reference in New Issue
Block a user