esh-pfi-infrastructure/docs/runbooks/nh3-dev-development-backup.md

nh3-dev ~/development — hourly off-box backup

Why this exists: nh3-dev is the dev box where agents do uncommitted work under ~/development/<project>/. That tree had no off-box backup, so a destructive mistake (a stray rm -rf on a working dir on 2026-07-12) had no safety net. This job closes that gap: an hourly, versioned, off-box snapshot of ~/development.

What it does

• Source: nh3-dev:~/development/ (lkraven's working dirs).
• Destination (off-box): nh3-nas:/volume1/Backup/nh3-dev-development/<YYYY-MM-DD_HHMM>/ — a timestamped dir per snapshot, over rsync-over-ssh (syncuser).
• Versioning: rsync --link-dest against the previous snapshot → unchanged files hardlink (share inodes, ~0 bytes); only changed files consume new space. latest symlink points at the newest snapshot.
• Retention (since 2026-10-02, Prime): the newest 48 hourly snapshots, plus the newest snapshot of each of the last 30 days, plus the newest of each of the last 12 ISO weeks (about 80 dirs at steady state; hardlinks keep the extra cost to changed files). Days and weeks count only those that have snapshots, so an outage does not eat the history. ~/.config/dev-backup/retention.py decides what to delete (repo copy: scripts/nh3-dev-development-backup-retention.py). It never selects a name that is not exactly YYYY-MM-DD_HHMM, and the NAS side refuses any path outside that pattern as a second guard. The prune runs chmod -R u+w before rm -rf: rsync copies a read-only source dir as read-only, and rm cannot unlink inside it. Each run logs retention prune: deleted <n>, <errors> error lines, <N> snapshots on the NAS (expected ... <K> kept); any error or N ≠ K exits 3, and a failed rsync exits 1, so either one leaves the unit failed (systemctl --user --failed). ⚠ Before 2026-10-01 the prune had no chmod and the run logged OK whatever happened: it failed silently from 2026-07-18 and left 1,740 husk dirs (each holding only the one 0555 dir), all removed 2026-10-01. History before 2026-09-30 is therefore gone; dailies and weeklies accumulate from 2026-10-02.
• Excludes: heavy reconstructable dirs (node_modules, .venv, venv, __pycache__, .pytest_cache, .mypy_cache, .ruff_cache, .cache, dist, build, .next, target, *.pyc) and secrets (.env, .env.*, *.pem, *.key, id_*, *.sqlite*). .git is kept (local commits/stashes = the uncommitted work that matters). Seed snapshot ≈ 11G; hourly deltas are MB-scale.

Where it lives (on nh3-dev)

• Script: ~/.config/dev-backup/dev-backup.sh (mirror committed at scripts/nh3-dev-development-backup.sh).
• systemd --user units: ~/.config/systemd/user/dev-backup.{service,timer} (OnCalendar=hourly, Persistent=true, linger on → fires without a login).
• Log: ~/.config/dev-backup/dev-backup.log.

systemctl --user list-timers dev-backup.timer   # next run
systemctl --user start dev-backup.service        # run now
tail -f ~/.config/dev-backup/dev-backup.log

Restore

Snapshots are plain dir trees — no special tool needed:

# list snapshots
ssh nh3-nas 'ls -1 /volume1/Backup/nh3-dev-development/'
# restore one file/dir from a chosen snapshot
rsync -a nh3-nas:/volume1/Backup/nh3-dev-development/<STAMP>/<proj>/<path> /tmp/restore/
# or pull a whole project back
rsync -a nh3-nas:/volume1/Backup/nh3-dev-development/latest/<proj>/ ~/development/<proj>/

Notes / future

• Not encrypted at rest (plaintext on the trusted internal NAS; secrets are excluded). Upgrade path: migrate to restic once a repo can be created on rest-server-nh3 (currently returns 404 on repo-create — likely append-only) or the Synology sftp subsystem is enabled (currently disabled → restic sftp fails).
• Off-box = off the nh3-dev VM (lands on nh3-nas, same NH3 site). Cross-site mirroring of this repo is a separate future layer.