# nh3-dev `~/development` — hourly off-box backup

**Why this exists:** nh3-dev is the dev box where agents do uncommitted work under
`~/development/<project>/`. That tree had **no off-box backup**, so a destructive
mistake (a stray `rm -rf` on a working dir on 2026-07-12) had no safety net. This
job closes that gap: an hourly, versioned, off-box snapshot of `~/development`.

## What it does

- **Source:** `nh3-dev:~/development/` (lkraven's working dirs).
- **Destination (off-box):** `nh3-nas:/volume1/Backup/nh3-dev-development/<YYYY-MM-DD_HHMM>/`
  — a timestamped dir per snapshot, over rsync-**over-ssh** (syncuser).
- **Versioning:** `rsync --link-dest` against the previous snapshot → unchanged
  files hardlink (share inodes, ~0 bytes); only changed files consume new space.
  `latest` symlink points at the newest snapshot.
- **Retention (since 2026-10-02, Prime):** the newest **48 hourly** snapshots, plus the newest
  snapshot of each of the last **30 days**, plus the newest of each of the last **12 ISO weeks**
  (about 80 dirs at steady state; hardlinks keep the extra cost to changed files). Days and weeks
  count only those that have snapshots, so an outage does not eat the history.
  `~/.config/dev-backup/retention.py` decides what to delete (repo copy:
  `scripts/nh3-dev-development-backup-retention.py`). It never selects a name that is not exactly
  `YYYY-MM-DD_HHMM`, and the NAS side refuses any path outside that pattern as a second guard.
  The prune runs `chmod -R u+w` before `rm -rf`: rsync copies a read-only source dir as read-only,
  and `rm` cannot unlink inside it. Each run logs
  `retention prune: deleted <n>, <errors> error lines, <N> snapshots on the NAS (expected ... <K> kept)`;
  any error or N ≠ K exits 3, and a failed rsync exits 1, so either one leaves the unit **failed**
  (`systemctl --user --failed`). ⚠ Before 2026-10-01 the prune had no chmod and the run logged OK
  whatever happened: it failed silently from 2026-07-18 and left 1,740 husk dirs (each holding only
  the one 0555 dir), all removed 2026-10-01. History before 2026-09-30 is therefore gone; dailies
  and weeklies accumulate from 2026-10-02.
- **Excludes:** heavy reconstructable dirs (`node_modules`, `.venv`, `venv`,
  `__pycache__`, `.pytest_cache`, `.mypy_cache`, `.ruff_cache`, `.cache`, `dist`,
  `build`, `.next`, `target`, `*.pyc`) and secrets (`.env`, `.env.*`, `*.pem`,
  `*.key`, `id_*`, `*.sqlite*`). **`.git` is kept** (local commits/stashes = the
  uncommitted work that matters). Seed snapshot ≈ **11G**; hourly deltas are MB-scale.

## Where it lives (on nh3-dev)

- Script: `~/.config/dev-backup/dev-backup.sh` (mirror committed at
  `scripts/nh3-dev-development-backup.sh`).
- systemd `--user` units: `~/.config/systemd/user/dev-backup.{service,timer}`
  (`OnCalendar=hourly`, `Persistent=true`, linger on → fires without a login).
- Log: `~/.config/dev-backup/dev-backup.log`.

```bash
systemctl --user list-timers dev-backup.timer   # next run
systemctl --user start dev-backup.service        # run now
tail -f ~/.config/dev-backup/dev-backup.log
```

## Restore

Snapshots are plain dir trees — no special tool needed:

```bash
# list snapshots
ssh nh3-nas 'ls -1 /volume1/Backup/nh3-dev-development/'
# restore one file/dir from a chosen snapshot
rsync -a nh3-nas:/volume1/Backup/nh3-dev-development/<STAMP>/<proj>/<path> /tmp/restore/
# or pull a whole project back
rsync -a nh3-nas:/volume1/Backup/nh3-dev-development/latest/<proj>/ ~/development/<proj>/
```

## Notes / future

- **Not encrypted at rest** (plaintext on the trusted internal NAS; secrets are
  excluded). Upgrade path: migrate to restic once a repo can be created on
  rest-server-nh3 (currently returns 404 on repo-create — likely append-only) or
  the Synology sftp subsystem is enabled (currently disabled → restic sftp fails).
- Off-box = off the nh3-dev VM (lands on nh3-nas, same NH3 site). Cross-site
  mirroring of this repo is a separate future layer.