docs(backup): hourly off-box ~/development backup to nh3-nas (runbook + script)

Adds the rsync --link-dest hourly snapshot job (nh3-dev:~/development ->
nh3-nas, 48-snapshot retention, secrets/build-dirs excluded) that closes the
no-off-box-backup gap exposed by the 2026-07-12 working-dir clobber. Script
mirrors the live ~/.config/dev-backup/dev-backup.sh; runbook covers restore.

docs/runbooks/nh3-dev-development-backup.md

@@ -0,0 +1,57 @@
+# nh3-dev `~/development` — hourly off-box backup
+
+**Why this exists:** nh3-dev is the dev box where agents do uncommitted work under
+`~/development/<project>/`. That tree had **no off-box backup**, so a destructive
+mistake (a stray `rm -rf` on a working dir on 2026-07-12) had no safety net. This
+job closes that gap: an hourly, versioned, off-box snapshot of `~/development`.
+
+## What it does
+
+- **Source:** `nh3-dev:~/development/` (lkraven's working dirs).
+- **Destination (off-box):** `nh3-nas:/volume1/Backup/nh3-dev-development/<YYYY-MM-DD_HHMM>/`
+  — a timestamped dir per snapshot, over rsync-**over-ssh** (syncuser).
+- **Versioning:** `rsync --link-dest` against the previous snapshot → unchanged
+  files hardlink (share inodes, ~0 bytes); only changed files consume new space.
+  `latest` symlink points at the newest snapshot.
+- **Retention:** newest **48** hourly snapshots (older pruned each run).
+- **Excludes:** heavy reconstructable dirs (`node_modules`, `.venv`, `venv`,
+  `__pycache__`, `.pytest_cache`, `.mypy_cache`, `.ruff_cache`, `.cache`, `dist`,
+  `build`, `.next`, `target`, `*.pyc`) and secrets (`.env`, `.env.*`, `*.pem`,
+  `*.key`, `id_*`, `*.sqlite*`). **`.git` is kept** (local commits/stashes = the
+  uncommitted work that matters). Seed snapshot ≈ **11G**; hourly deltas are MB-scale.
+
+## Where it lives (on nh3-dev)
+
+- Script: `~/.config/dev-backup/dev-backup.sh` (mirror committed at
+  `scripts/nh3-dev-development-backup.sh`).
+- systemd `--user` units: `~/.config/systemd/user/dev-backup.{service,timer}`
+  (`OnCalendar=hourly`, `Persistent=true`, linger on → fires without a login).
+- Log: `~/.config/dev-backup/dev-backup.log`.
+
+```bash
+systemctl --user list-timers dev-backup.timer   # next run
+systemctl --user start dev-backup.service        # run now
+tail -f ~/.config/dev-backup/dev-backup.log
+```
+
+## Restore
+
+Snapshots are plain dir trees — no special tool needed:
+
+```bash
+# list snapshots
+ssh nh3-nas 'ls -1 /volume1/Backup/nh3-dev-development/'
+# restore one file/dir from a chosen snapshot
+rsync -a nh3-nas:/volume1/Backup/nh3-dev-development/<STAMP>/<proj>/<path> /tmp/restore/
+# or pull a whole project back
+rsync -a nh3-nas:/volume1/Backup/nh3-dev-development/latest/<proj>/ ~/development/<proj>/
+```
+
+## Notes / future
+
+- **Not encrypted at rest** (plaintext on the trusted internal NAS; secrets are
+  excluded). Upgrade path: migrate to restic once a repo can be created on
+  rest-server-nh3 (currently returns 404 on repo-create — likely append-only) or
+  the Synology sftp subsystem is enabled (currently disabled → restic sftp fails).
+- Off-box = off the nh3-dev VM (lands on nh3-nas, same NH3 site). Cross-site
+  mirroring of this repo is a separate future layer.

scripts/nh3-dev-development-backup.sh

@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+# Hourly off-box snapshot of ~/development -> nh3-nas via rsync --link-dest
+# hardlink snapshots. Penance for the 2026-07-12 soong-lab clobber: uncommitted
+# dev work now has an hourly, versioned, off-box safety net. Secrets + heavy
+# reconstructable dirs are excluded. Snapshots are timestamped dirs on the NAS;
+# unchanged files hardlink to the previous snapshot (space-efficient). Retention:
+# newest 48 hourly snapshots.
+set -uo pipefail
+
+SRC="$HOME/development/"
+DEST_HOST="nh3-nas"
+DEST_BASE="/volume1/Backup/nh3-dev-development"
+STAMP="$(date +%Y-%m-%d_%H%M)"
+LOG="$HOME/.config/dev-backup/dev-backup.log"
+
+exec >>"$LOG" 2>&1
+echo "=== $(date -Is) snapshot $STAMP start ==="
+
+# previous snapshot for hardlink dedup
+PREV="$(ssh -o ConnectTimeout=15 -o BatchMode=yes "$DEST_HOST" "ls -1d $DEST_BASE/20* 2>/dev/null | sort | tail -1" || true)"
+LINKDEST=()
+[ -n "$PREV" ] && LINKDEST=(--link-dest="$PREV")
+echo "link-dest: ${PREV:-<none, first full snapshot>}"
+
+ssh -o BatchMode=yes "$DEST_HOST" "mkdir -p '$DEST_BASE/$STAMP'"
+
+rsync -a --delete --numeric-ids \
+  --exclude='node_modules/' --exclude='.venv/' --exclude='venv/' --exclude='__pycache__/' \
+  --exclude='.pytest_cache/' --exclude='.mypy_cache/' --exclude='.ruff_cache/' --exclude='.cache/' \
+  --exclude='dist/' --exclude='build/' --exclude='.next/' --exclude='target/' --exclude='*.pyc' \
+  --exclude='.env' --exclude='.env.*' --exclude='*.pem' --exclude='*.key' --exclude='id_*' \
+  --exclude='*.sqlite' --exclude='*.sqlite3' --exclude='*.db-wal' --exclude='*.db-shm' \
+  "${LINKDEST[@]}" \
+  "$SRC" "$DEST_HOST:$DEST_BASE/$STAMP/"
+RC=$?
+echo "rsync rc=$RC"
+
+# rc 0 = ok; rc 24 = some files vanished mid-transfer (benign for a live tree)
+if [ "$RC" -eq 0 ] || [ "$RC" -eq 24 ]; then
+  ssh -o BatchMode=yes "$DEST_HOST" "ln -sfn '$DEST_BASE/$STAMP' '$DEST_BASE/latest'"
+  # retention: keep newest 48 hourly snapshots
+  ssh -o BatchMode=yes "$DEST_HOST" "ls -1d $DEST_BASE/20* 2>/dev/null | sort | head -n -48 | xargs -r rm -rf"
+  echo "=== $(date -Is) snapshot $STAMP OK (rc=$RC) ==="
+else
+  echo "=== $(date -Is) snapshot $STAMP FAILED rc=$RC — keeping partial for inspection ==="
+fi