diff --git a/docs/runbooks/nh3-dev-development-backup.md b/docs/runbooks/nh3-dev-development-backup.md new file mode 100644 index 0000000..cf462a7 --- /dev/null +++ b/docs/runbooks/nh3-dev-development-backup.md @@ -0,0 +1,57 @@ +# nh3-dev `~/development` — hourly off-box backup + +**Why this exists:** nh3-dev is the dev box where agents do uncommitted work under +`~/development//`. That tree had **no off-box backup**, so a destructive +mistake (a stray `rm -rf` on a working dir on 2026-07-12) had no safety net. This +job closes that gap: an hourly, versioned, off-box snapshot of `~/development`. + +## What it does + +- **Source:** `nh3-dev:~/development/` (lkraven's working dirs). +- **Destination (off-box):** `nh3-nas:/volume1/Backup/nh3-dev-development//` + — a timestamped dir per snapshot, over rsync-**over-ssh** (syncuser). +- **Versioning:** `rsync --link-dest` against the previous snapshot → unchanged + files hardlink (share inodes, ~0 bytes); only changed files consume new space. + `latest` symlink points at the newest snapshot. +- **Retention:** newest **48** hourly snapshots (older pruned each run). +- **Excludes:** heavy reconstructable dirs (`node_modules`, `.venv`, `venv`, + `__pycache__`, `.pytest_cache`, `.mypy_cache`, `.ruff_cache`, `.cache`, `dist`, + `build`, `.next`, `target`, `*.pyc`) and secrets (`.env`, `.env.*`, `*.pem`, + `*.key`, `id_*`, `*.sqlite*`). **`.git` is kept** (local commits/stashes = the + uncommitted work that matters). Seed snapshot ≈ **11G**; hourly deltas are MB-scale. + +## Where it lives (on nh3-dev) + +- Script: `~/.config/dev-backup/dev-backup.sh` (mirror committed at + `scripts/nh3-dev-development-backup.sh`). +- systemd `--user` units: `~/.config/systemd/user/dev-backup.{service,timer}` + (`OnCalendar=hourly`, `Persistent=true`, linger on → fires without a login). +- Log: `~/.config/dev-backup/dev-backup.log`. + +```bash +systemctl --user list-timers dev-backup.timer # next run +systemctl --user start dev-backup.service # run now +tail -f ~/.config/dev-backup/dev-backup.log +``` + +## Restore + +Snapshots are plain dir trees — no special tool needed: + +```bash +# list snapshots +ssh nh3-nas 'ls -1 /volume1/Backup/nh3-dev-development/' +# restore one file/dir from a chosen snapshot +rsync -a nh3-nas:/volume1/Backup/nh3-dev-development/// /tmp/restore/ +# or pull a whole project back +rsync -a nh3-nas:/volume1/Backup/nh3-dev-development/latest// ~/development// +``` + +## Notes / future + +- **Not encrypted at rest** (plaintext on the trusted internal NAS; secrets are + excluded). Upgrade path: migrate to restic once a repo can be created on + rest-server-nh3 (currently returns 404 on repo-create — likely append-only) or + the Synology sftp subsystem is enabled (currently disabled → restic sftp fails). +- Off-box = off the nh3-dev VM (lands on nh3-nas, same NH3 site). Cross-site + mirroring of this repo is a separate future layer. diff --git a/scripts/nh3-dev-development-backup.sh b/scripts/nh3-dev-development-backup.sh new file mode 100755 index 0000000..9949056 --- /dev/null +++ b/scripts/nh3-dev-development-backup.sh @@ -0,0 +1,46 @@ +#!/usr/bin/env bash +# Hourly off-box snapshot of ~/development -> nh3-nas via rsync --link-dest +# hardlink snapshots. Penance for the 2026-07-12 soong-lab clobber: uncommitted +# dev work now has an hourly, versioned, off-box safety net. Secrets + heavy +# reconstructable dirs are excluded. Snapshots are timestamped dirs on the NAS; +# unchanged files hardlink to the previous snapshot (space-efficient). Retention: +# newest 48 hourly snapshots. +set -uo pipefail + +SRC="$HOME/development/" +DEST_HOST="nh3-nas" +DEST_BASE="/volume1/Backup/nh3-dev-development" +STAMP="$(date +%Y-%m-%d_%H%M)" +LOG="$HOME/.config/dev-backup/dev-backup.log" + +exec >>"$LOG" 2>&1 +echo "=== $(date -Is) snapshot $STAMP start ===" + +# previous snapshot for hardlink dedup +PREV="$(ssh -o ConnectTimeout=15 -o BatchMode=yes "$DEST_HOST" "ls -1d $DEST_BASE/20* 2>/dev/null | sort | tail -1" || true)" +LINKDEST=() +[ -n "$PREV" ] && LINKDEST=(--link-dest="$PREV") +echo "link-dest: ${PREV:-}" + +ssh -o BatchMode=yes "$DEST_HOST" "mkdir -p '$DEST_BASE/$STAMP'" + +rsync -a --delete --numeric-ids \ + --exclude='node_modules/' --exclude='.venv/' --exclude='venv/' --exclude='__pycache__/' \ + --exclude='.pytest_cache/' --exclude='.mypy_cache/' --exclude='.ruff_cache/' --exclude='.cache/' \ + --exclude='dist/' --exclude='build/' --exclude='.next/' --exclude='target/' --exclude='*.pyc' \ + --exclude='.env' --exclude='.env.*' --exclude='*.pem' --exclude='*.key' --exclude='id_*' \ + --exclude='*.sqlite' --exclude='*.sqlite3' --exclude='*.db-wal' --exclude='*.db-shm' \ + "${LINKDEST[@]}" \ + "$SRC" "$DEST_HOST:$DEST_BASE/$STAMP/" +RC=$? +echo "rsync rc=$RC" + +# rc 0 = ok; rc 24 = some files vanished mid-transfer (benign for a live tree) +if [ "$RC" -eq 0 ] || [ "$RC" -eq 24 ]; then + ssh -o BatchMode=yes "$DEST_HOST" "ln -sfn '$DEST_BASE/$STAMP' '$DEST_BASE/latest'" + # retention: keep newest 48 hourly snapshots + ssh -o BatchMode=yes "$DEST_HOST" "ls -1d $DEST_BASE/20* 2>/dev/null | sort | head -n -48 | xargs -r rm -rf" + echo "=== $(date -Is) snapshot $STAMP OK (rc=$RC) ===" +else + echo "=== $(date -Is) snapshot $STAMP FAILED rc=$RC — keeping partial for inspection ===" +fi