diff --git a/configs/restic/ana-ml2/README.md b/configs/restic/ana-ml2/README.md new file mode 100644 index 0000000..a875f6d --- /dev/null +++ b/configs/restic/ana-ml2/README.md @@ -0,0 +1,151 @@ +# restic / ana-ml2 + +**Bare metal** — ana-ml2 runs directly on a server chassis, not on any +Proxmox hypervisor, so vzdump doesn't cover it. This is the sole backup +for the host's configuration and is the highest-stakes backup target in +the fleet. + +## What's backed up + +| Path | Purpose | +|---|---| +| `/opt/docker` | Compose files + config bind mounts (~110 MB) | +| `/etc` | Host config — systemd, ssh, chrony, apparmor, apt, etc. | +| `/root` | Root's ad-hoc scripts, shell history, ssh keys | +| `/var/lib/docker/volumes` | Named volumes used by running stacks (small — models live elsewhere) | + +## What's **not** backed up (by design) + +- **`/tank/*`** — model caches (HF hub, GGUFs, ComfyUI checkpoints, + Kokoro voices, Vibevoice weights, Parakeet weights). Regenerable from + upstream; storing them would blow the repo size budget. +- `/root/.cache`, `/root/.npm`, shell-noise caches +- Any `*.log`, `*.pid`, `.../logs/` directories + +## Stacks on this host + +All inference/AI-adjacent — none use relational databases, so no +pre-backup DB hook is required. Contrast with `configs/restic/ana-docker/` +where synapse/seafile/vaultwarden DB dumps run first. + +- `llama-swap` — GGUF swapper (llama.cpp) +- `vllm-qwen3` — embedding + rerank +- `comfyui`, `kokoro`, `parakeet`, `vibevoice` +- `beszel-agent-ana`, `dozzle-agent-ana`, `dockge` + +## Deploy (one-time setup) + +### 1. Add the client entry to the Anaheim rest-server .htpasswd + +On ana-docker (where rest-server-ana runs): + +```bash +ssh -t ana-docker ' + cd /opt/docker/data/rest-server-ana && # wherever the data dir is + sudo htpasswd -B /data/.htpasswd ana-ml2 && # prompts for a new password + docker restart rest-server +' +``` + +Store the password where ana-ml2 can grab it. This is the **HTTP basic +auth password**, not the restic repo passphrase — two separate secrets. + +### 2. Create the restic repo + +On ana-ml2, with a freshly generated encryption passphrase: + +```bash +ssh -t ana-ml2 ' + sudo install -d -o root -g root -m 0700 /etc/restic /var/lib/restic && + echo "RESTIC_REPOSITORY=rest:http://ana-ml2:@10.250.50.70:8000/ana-ml2/" | sudo tee /etc/restic/restic.env > /dev/null && + sudo chmod 600 /etc/restic/restic.env && + + # Generate a fresh passphrase (print it to stdout exactly once — store + # it in your password manager immediately; losing it = losing this + # host´s entire backup history). + openssl rand -base64 48 | sudo tee /etc/restic/password > /dev/null && + sudo chmod 600 /etc/restic/password && + + sudo env $(cat /etc/restic/restic.env) \ + RESTIC_PASSWORD_FILE=/etc/restic/password \ + restic init +' +``` + +### 3. Install resticprofile on ana-ml2 + +The host has `restic` but not `resticprofile`. From the resticprofile +docs: + +```bash +ssh -t ana-ml2 ' + curl -sfL https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh | sudo sh -s -- -b /usr/local/bin + /usr/local/bin/resticprofile --version +' +``` + +### 4. Deploy the profile + +```bash +scp configs/restic/ana-ml2/profiles.yaml ana-ml2:/tmp/profiles.yaml +ssh -t ana-ml2 ' + sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml && + rm /tmp/profiles.yaml +' +``` + +### 5. Generate + enable the systemd timers + +```bash +ssh -t ana-ml2 ' + sudo resticprofile --config /etc/restic/profiles.yaml schedule --all && + systemctl list-timers "resticprofile*" +' +``` + +`resticprofile schedule` writes the unit files into `/etc/systemd/system/` +and enables them. You should see three timers: backup, forget, check. + +### 6. Fire a manual backup to verify + +```bash +ssh -t ana-ml2 'sudo resticprofile --config /etc/restic/profiles.yaml backup --verbose' +``` + +Expected output: a fresh snapshot id and a line summarizing added data +size + file count. + +Cross-check from the hub: + +```bash +ssh ana-docker 'docker exec backrest ls /repos/ana-ml2/' +``` + +Or in Backrest UI (http://10.250.50.70:9898), `ana-ml2` shows up as a new +repo with one snapshot. + +## Ongoing + +Timers run at 01:00 / 03:00 / Sunday 05:00 (matching the rest of the +fleet). Monitor via the `schedule-log` entries or Backrest's web UI. + +## Restoring + +```bash +# List snapshots +ssh -t ana-ml2 'sudo resticprofile --config /etc/restic/profiles.yaml snapshots' + +# Restore a single path +ssh -t ana-ml2 'sudo resticprofile --config /etc/restic/profiles.yaml restore --target /tmp/restore latest --path /opt/docker' +``` + +## Gotchas specific to this host + +- **Large `/tank` is deliberately excluded** — don't edit source paths + to include it without a plan for the ~TB of model data. +- **Repo passphrase is irreplaceable.** Losing `/etc/restic/password` + without a copy elsewhere = losing every snapshot in the `ana-ml2` + repo. Store a copy in your password manager on day one. +- **`/var/lib/docker/volumes` assumes docker uses the default data + root.** If you ever switch to a custom dockerd data-root, update the + source list. diff --git a/configs/restic/ana-ml2/profiles.yaml b/configs/restic/ana-ml2/profiles.yaml new file mode 100644 index 0000000..f5e7209 --- /dev/null +++ b/configs/restic/ana-ml2/profiles.yaml @@ -0,0 +1,82 @@ +# resticprofile config for ana-ml2. +# +# **Bare metal** — not protected by any Proxmox vzdump. This is the only +# backup for this host's config/state, so it's the highest-stakes repo in +# the fleet. Writes to the Anaheim rest-server at 10.250.50.70 as user +# `ana-ml2`. The full REST URL (with HTTP basic auth creds) lives in +# /etc/restic/restic.env. The client-side encryption passphrase lives in +# /etc/restic/password. +# +# Notable exclusions: +# - /tank/* is NOT in source paths. That's ~TB of model weights (HF +# caches, llama.cpp GGUFs, ComfyUI models, etc.) — all regenerable +# from upstream. Backing them up would blow the repo size budget. +# - No DB dumps needed. None of the stacks on this host (llama-swap, +# vllm-qwen3, comfyui, kokoro, parakeet, vibevoice, beszel-agent, +# dozzle-agent, dockge) store relational data. + +version: "1" + +global: + priority: low + ionice: true + ionice-class: 2 + ionice-level: 7 + min-memory: 100 + +default: + env-file: /etc/restic/restic.env # RESTIC_REPOSITORY=rest:http://user:pw@… + env: + RESTIC_PASSWORD_FILE: /etc/restic/password + initialize: false # repo created manually by `restic init` + lock: /var/lock/restic-ana-ml2.lock + + backup: + verbose: 1 + run-after: + - date +%s > /var/lib/restic/last-success + source: + - /opt/docker # compose files + host-level configs — small (~100MB) + - /etc # host config (systemd units, chrony, apparmor, ssh, etc.) + - /root # root shell history, ssh keys, any ad-hoc scripts + - /var/lib/docker/volumes # named volumes (small; models live on /tank, not here) + exclude: + # Docker internals we never want in a backup + - /var/lib/docker/volumes/backingFsBlockDev + - /var/lib/docker/volumes/metadata.db + # Ephemeral / regenerable junk + - /opt/docker/compose/*/logs + - "**/*.log" + - "**/*.log.*" + - "**/*.pid" + # Root's noise — shell caches, tmp + - /root/.cache + - /root/.local/share/Trash + - /root/.npm + - /root/.python_history + tag: + - host:ana-ml2 + - site:ana + - fleet:pfi + schedule: "*-*-* 01:00:00" + schedule-permission: system + schedule-log: /var/log/restic-backup.log + + forget: + keep-daily: 7 + keep-weekly: 4 + keep-monthly: 12 + keep-yearly: 3 + # NOTE: no `prune: true` — rest-server runs with --append-only. + # See README.md "Prune ceremony". + tag: + - host:ana-ml2 + schedule: "*-*-* 03:00:00" + schedule-permission: system + schedule-log: /var/log/restic-forget.log + + check: + read-data-subset: 10% + schedule: "Sun *-*-* 05:00:00" + schedule-permission: system + schedule-log: /var/log/restic-check.log