diff --git a/configs/restic/irv-ml1/README.md b/configs/restic/irv-ml1/README.md new file mode 100644 index 0000000..3ec4e22 --- /dev/null +++ b/configs/restic/irv-ml1/README.md @@ -0,0 +1,180 @@ +# restic / irv-ml1 + +**Bare metal, remote site** — irv-ml1 runs directly on a server chassis +at Irvine, reachable only via WireGuard tunnel from NH3. No vzdump +coverage. Analog of `ana-ml2`, but targets `rest-server-nh3` (not +`rest-server-ana`) because NH3 is the WG termination endpoint — keeps +the backup traffic on the same side of the tunnel. + +## What's backed up + +| Path | Purpose | +|---|---| +| `/opt/docker` | Compose files + config bind mounts | +| `/etc` | Host config — systemd, ssh, chrony, apparmor, **wg configs**, apt | +| `/root` | Root's ad-hoc scripts, shell history, ssh keys | +| `/home/lkraven` | Admin home — dotfiles, dev scripts, small configs | +| `/var/lib/docker/volumes` | Named volumes from dockerized stacks | +| `/worktank/comfyui/basedir/user` | ComfyUI workflows + per-user settings | +| `/worktank/comfyui/basedir/custom_nodes` | Hand-installed ComfyUI extensions | +| `/worktank/comfyui/basedir/input` | User-uploaded source images for workflows | +| `/worktank/cosyvoice/voices` | Cloned speaker profiles — reproducing them needs the original reference audio | + +## What's **not** backed up (by design) + +- **`/storetank/*`** — ~1.8 TB model weights. Regenerable from + HuggingFace Hub / upstream repos. Do NOT add to source without a + plan for the ~TB of data. +- **`/worktank/*` except the selective dirs above** — the rest is + bulk, regenerable, or disposable: + - `/worktank/comfyui/basedir/{models,output,temp}` — bulk weights + + generated outputs + internal scratch + - `/worktank/comfyui/run` — ~8 GB disposable Python venv + pip + cache (container runtime for comfyui) + - `/worktank/parakeet/models` — HF cache, re-downloads + - `/worktank/cosyvoice/{input,output}` — scratch audio +- **Native AI stacks under `/opt/{ComfyUI,sillytavern,llama.cpp,...}`** + — git clones + pip envs. Regenerable. Only `/opt/docker` (fleet + convention for containerized stacks) is in the source list. +- **`/home/llmuser`** and **`/home/sduser`** — belong on `/storetank` + / regenerable; kept out via explicit exclude. +- Home-dir `.cache/`, `.npm`, HF cache, conda envs, virtualenvs, + `.cargo/registry`, Rust toolchain, `*.log`, `*.pid` +- Docker internals (`backingFsBlockDev`, `metadata.db`) + +## Deploy (one-time setup) + +### 1. Create rest-server-nh3 credentials for irv-ml1 + +On `nh3-nas` (where rest-server-nh3's data dir lives — `/volume1/Backup/restic/`), +add an htpasswd entry for user `irv-ml1`: + +```bash +# Pick a strong password, store in password manager BEFORE running this +NEW_PW='' + +ssh -t nh3-nas ' + # rest-server-nh3 .htpasswd path under /volume1/Backup/restic/ + HTPATH=/volume1/Backup/restic/.htpasswd + sudo docker run --rm httpd:2.4-alpine htpasswd -nbB irv-ml1 "'"$NEW_PW"'" \ + | sudo tee -a "$HTPATH" > /dev/null + sudo tail -2 "$HTPATH" +' +# Restart rest-server-nh3 so it picks up the new credential +ssh -t nh3-nas 'sudo docker restart rest-server-nh3' +``` + +### 2. Install secret files on irv-ml1 + +```bash +ssh -t irv-ml1 ' + sudo install -d -o root -g root -m 0700 /etc/restic /var/lib/restic && + sudo tee /etc/restic/restic.env > /dev/null <@10.100.50.50:8000/irv-ml1/ +EOF + sudo chmod 600 /etc/restic/restic.env +' +``` + +Then install the repo passphrase (generate a fresh one — this is a +new repo, no prior passphrase to match): + +```bash +ssh -t irv-ml1 'sudo bash -c " + openssl rand -base64 48 | tr -d \"\\n\" > /etc/restic/password + chmod 600 /etc/restic/password + cat /etc/restic/password # copy this to your password manager NOW + echo +"' +``` + +**Copy the printed passphrase to your password manager before doing +anything else.** It prints exactly once. Lose it = lose the repo. + +### 3. Initialize the repo + +```bash +ssh -t irv-ml1 ' + sudo bash -c " + set -a; . /etc/restic/restic.env; set +a + RESTIC_PASSWORD_FILE=/etc/restic/password restic init + " +' +``` + +Expected: `created restic repository at rest:http://...` + +### 4. Install resticprofile on irv-ml1 + +```bash +ssh -t irv-ml1 ' + curl -sfL https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh | sudo sh -s -- -b /usr/local/bin + /usr/local/bin/resticprofile --version +' +``` + +### 5. Deploy the profile + +```bash +scp configs/restic/irv-ml1/profiles.yaml irv-ml1:/tmp/profiles.yaml +ssh -t irv-ml1 ' + sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml && + rm /tmp/profiles.yaml +' +``` + +### 6. Generate + enable the systemd timers + +```bash +ssh -t irv-ml1 ' + sudo resticprofile --config /etc/restic/profiles.yaml schedule --all && + systemctl list-timers "resticprofile*" +' +``` + +### 7. First manual backup + +```bash +ssh -t irv-ml1 'sudo resticprofile --config /etc/restic/profiles.yaml backup --verbose' +``` + +Expect the first run to be a few hundred MB (host config + home dirs; +nothing from /storetank or /opt native stacks). + +## WireGuard-specific gotcha + +The backup traverses the WG tunnel. If the tunnel drops, the timer +will fail — restic doesn't auto-retry a missed window, and the timer +won't catch up until the next `01:00:00` fires. Consider: + +- Monitor WG health separately (Beszel agent ping of 10.100.50.50) +- Expect more "failed backup" noise if WG is flaky; investigate + the tunnel, not restic + +## Ongoing + +Timers fire at 01:00 / Sunday 05:00, matching the rest of the fleet. + +## Restoring + +Same pattern as ana-ml2 — see `configs/restic/ana-ml2/README.md` for +the full flow. + +## Gotchas specific to this host + +- **`/storetank` excluded entirely; `/worktank` included selectively.** + Stacks that place user state under `/worktank//` (comfyui, + parakeet, cosyvoice) are cherry-picked by subtree; bulk/regenerable + dirs stay out. See "What's backed up / not backed up" above. +- **Native AI stacks under `/opt/`** (ComfyUI, SillyTavern, etc.) are + also NOT backed up. If the user fine-tunes config/weights inside + those trees and wants to preserve specific paths, add them to the + source list explicitly. (The dockerized ComfyUI at + `/opt/docker/compose/comfyui` **is** covered via `/opt/docker`; only + the native `/opt/ComfyUI` install is skipped.) +- **Backup traffic crosses the WG tunnel.** Latency is higher than + in-LAN restic clients; first full backup may take longer than an + equivalent ana-side host. +- **Repo passphrase is irreplaceable.** Losing `/etc/restic/password` + without a copy elsewhere = losing every snapshot. Password manager + on day one. diff --git a/configs/restic/irv-ml1/profiles.yaml b/configs/restic/irv-ml1/profiles.yaml new file mode 100644 index 0000000..7b67ac3 --- /dev/null +++ b/configs/restic/irv-ml1/profiles.yaml @@ -0,0 +1,138 @@ +# resticprofile config for irv-ml1. +# +# **Bare metal, remote site.** Not protected by any Proxmox vzdump — this +# is the sole backup for this host. Reachable only via WireGuard +# (10.100.79.3 is the WG tunnel IP). Backups target rest-server-nh3 at +# 10.100.50.50 because that's the closest endpoint through the WG tunnel; +# cross-site to ana-docker would double-hop the WAN. +# +# Notable exclusions: +# - /storetank/* — ~1.8 TB of model weights (ComfyUI checkpoints, HF hub, +# llama.cpp GGUFs, etc.). All regenerable from upstream. Do NOT add. +# - /worktank/* — *selectively* included. Docker stacks on this host now +# place user state under /worktank// (comfyui, parakeet, +# cosyvoice). Sources are explicit per-subtree; bulk/scratch/run +# dirs stay out. Rule of thumb: precious human-generated data in, +# downloadable weights + regenerable outputs + disposable runtime +# out. +# - Native AI toolchain under /opt/{ComfyUI,sillytavern,stablediffusion, +# llama.cpp,kokoro,...} — git clones / pip envs, regenerable from +# source. Only /opt/docker is backed up (compose files + configs, +# once Docker is installed following the fleet convention). +# +# No DB dumps needed — this host's workloads don't store relational data. + +version: "1" + +global: + priority: low + ionice: true + ionice-class: 2 + ionice-level: 7 + min-memory: 100 + +default: + env-file: /etc/restic/restic.env # RESTIC_REPOSITORY=rest:http://user:pw@10.100.50.50:8000/irv-ml1/ + env: + RESTIC_PASSWORD_FILE: /etc/restic/password + initialize: false # repo created manually by `restic init` + lock: /var/lock/restic-irv-ml1.lock + + backup: + verbose: 1 + run-after: + - date +%s > /var/lib/restic/last-success + source: + - /opt/docker # compose files + bind-mount configs + - /etc # host config — systemd, chrony, apparmor, ssh, wg + - /root # root's ad-hoc scripts, shell history, ssh keys + - /home/lkraven # primary admin home dir — dotfiles, dev scripts, small configs + - /var/lib/docker/volumes # named volumes from dockerized stacks + # Selective /worktank inclusions (the rest of /worktank stays out): + - /worktank/comfyui/basedir/user # ComfyUI workflows + settings + - /worktank/comfyui/basedir/custom_nodes # hand-installed ComfyUI extensions + - /worktank/comfyui/basedir/input # user-uploaded source images for workflows + - /worktank/cosyvoice/voices # cloned speaker profiles — reproducing them needs the original reference audio + # Deliberately NOT in source (see "What's not backed up" in README): + # /home/llmuser # ~380 GB of model weights / training outputs + # # (contents belong on /storetank; regenerable from HF) + # /home/sduser # Stable Diffusion user state (~2.6 GB); scope-check + # # before adding — likely regenerable + # /worktank/comfyui/basedir/{models,output,temp} # bulk + regenerable + # /worktank/comfyui/run # ~8 GB disposable venv + # /worktank/parakeet/models # HF cache, regenerable + # /worktank/cosyvoice/{input,output} # scratch + exclude: + # Explicit protection: even if source is ever changed to /home, skip + # llmuser entirely — the only reason it's 380 GB is AI artifacts that + # belong on /storetank or HF hub, not in a config-backup repo. + - /home/llmuser + - /home/sduser + # Defense-in-depth: even if `source` is ever broadened to /worktank + # wholesale, these must never enter the repo. + - /worktank/comfyui/basedir/models + - /worktank/comfyui/basedir/output + - /worktank/comfyui/basedir/temp + - /worktank/comfyui/run + - /worktank/parakeet/models + - /worktank/cosyvoice/input + - /worktank/cosyvoice/output + # Docker internals we never want in a backup + - /var/lib/docker/volumes/backingFsBlockDev + - /var/lib/docker/volumes/metadata.db + # Ephemeral / regenerable junk + - /opt/docker/compose/*/logs + - "**/*.log" + - "**/*.log.*" + - "**/*.pid" + # User-cache / toolchain noise under home dirs (saves ~5.8 GB on /home/lkraven alone) + - /home/*/.cache + - /home/*/.npm + - /home/*/.local/share/Trash + - /home/*/.cargo/registry + - /home/*/.cargo/git + - /home/*/.rustup + - /home/*/huggingface + - /home/*/.cache/huggingface + - /home/*/conda + - /home/*/anaconda3 + - /home/*/miniconda3 + - /home/*/.conda + - /home/*/micromamba + # Common Python virtualenv noise + - "**/__pycache__" + - "**/.venv" + - "**/venv" + - "**/env" + - "**/.pytest_cache" + # Root's noise + - /root/.cache + - /root/.local/share/Trash + - /root/.npm + - /root/.python_history + tag: + - host:irv-ml1 + - site:irv + - fleet:pfi + schedule: "*-*-* 01:00:00" + schedule-permission: system + schedule-log: /var/log/restic-backup.log + + forget: + keep-daily: 7 + keep-weekly: 4 + keep-monthly: 12 + keep-yearly: 3 + # NOTE: no `prune: true` — rest-server runs with --append-only. + # See README.md "Prune ceremony". + tag: + - host:irv-ml1 + # Schedule removed at deploy: forget against --append-only rest-server + # always fails (delete ops blocked). Run manually during the prune + # ceremony when --append-only is temporarily off. + + check: + read-data-subset: 10% + schedule: "Sun *-*-* 05:00:00" + schedule-permission: system + schedule-log: /var/log/restic-check.log