restic: ana-ml2 profile — cover the only bare-metal host in the fleet

ana-ml2 is not on any Proxmox hypervisor, so vzdump doesn't touch it.
This closes the biggest single backup gap per the 2026-04-20 pipeline
audit.

Sources: /opt/docker (~110 MB), /etc, /root, /var/lib/docker/volumes.
Excludes /tank/* (model weights — regenerable from Hugging Face and
would blow repo size budget). No pre-backup DB hook — none of the
llama-swap / vllm / comfyui / kokoro / parakeet / vibevoice stacks
use relational databases.

README walks through the one-time setup: rest-server .htpasswd entry,
restic init with fresh passphrase, resticprofile install, systemd timer
generation, verification against the Backrest UI.

configs/restic/ana-ml2/README.md

@@ -0,0 +1,151 @@
+# restic / ana-ml2
+
+**Bare metal** — ana-ml2 runs directly on a server chassis, not on any
+Proxmox hypervisor, so vzdump doesn't cover it. This is the sole backup
+for the host's configuration and is the highest-stakes backup target in
+the fleet.
+
+## What's backed up
+
+| Path | Purpose |
+|---|---|
+| `/opt/docker` | Compose files + config bind mounts (~110 MB) |
+| `/etc` | Host config — systemd, ssh, chrony, apparmor, apt, etc. |
+| `/root` | Root's ad-hoc scripts, shell history, ssh keys |
+| `/var/lib/docker/volumes` | Named volumes used by running stacks (small — models live elsewhere) |
+
+## What's **not** backed up (by design)
+
+- **`/tank/*`** — model caches (HF hub, GGUFs, ComfyUI checkpoints,
+  Kokoro voices, Vibevoice weights, Parakeet weights). Regenerable from
+  upstream; storing them would blow the repo size budget.
+- `/root/.cache`, `/root/.npm`, shell-noise caches
+- Any `*.log`, `*.pid`, `.../logs/` directories
+
+## Stacks on this host
+
+All inference/AI-adjacent — none use relational databases, so no
+pre-backup DB hook is required. Contrast with `configs/restic/ana-docker/`
+where synapse/seafile/vaultwarden DB dumps run first.
+
+- `llama-swap` — GGUF swapper (llama.cpp)
+- `vllm-qwen3` — embedding + rerank
+- `comfyui`, `kokoro`, `parakeet`, `vibevoice`
+- `beszel-agent-ana`, `dozzle-agent-ana`, `dockge`
+
+## Deploy (one-time setup)
+
+### 1. Add the client entry to the Anaheim rest-server .htpasswd
+
+On ana-docker (where rest-server-ana runs):
+
+```bash
+ssh -t ana-docker '
+  cd /opt/docker/data/rest-server-ana &&   # wherever the data dir is
+  sudo htpasswd -B /data/.htpasswd ana-ml2 &&   # prompts for a new password
+  docker restart rest-server
+'
+```
+
+Store the password where ana-ml2 can grab it. This is the **HTTP basic
+auth password**, not the restic repo passphrase — two separate secrets.
+
+### 2. Create the restic repo
+
+On ana-ml2, with a freshly generated encryption passphrase:
+
+```bash
+ssh -t ana-ml2 '
+  sudo install -d -o root -g root -m 0700 /etc/restic /var/lib/restic &&
+  echo "RESTIC_REPOSITORY=rest:http://ana-ml2:<htpasswd-password>@10.250.50.70:8000/ana-ml2/" | sudo tee /etc/restic/restic.env > /dev/null &&
+  sudo chmod 600 /etc/restic/restic.env &&
+
+  # Generate a fresh passphrase (print it to stdout exactly once — store
+  # it in your password manager immediately; losing it = losing this
+  # host´s entire backup history).
+  openssl rand -base64 48 | sudo tee /etc/restic/password > /dev/null &&
+  sudo chmod 600 /etc/restic/password &&
+
+  sudo env $(cat /etc/restic/restic.env) \
+       RESTIC_PASSWORD_FILE=/etc/restic/password \
+       restic init
+'
+```
+
+### 3. Install resticprofile on ana-ml2
+
+The host has `restic` but not `resticprofile`. From the resticprofile
+docs:
+
+```bash
+ssh -t ana-ml2 '
+  curl -sfL https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh | sudo sh -s -- -b /usr/local/bin
+  /usr/local/bin/resticprofile --version
+'
+```
+
+### 4. Deploy the profile
+
+```bash
+scp configs/restic/ana-ml2/profiles.yaml ana-ml2:/tmp/profiles.yaml
+ssh -t ana-ml2 '
+  sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml &&
+  rm /tmp/profiles.yaml
+'
+```
+
+### 5. Generate + enable the systemd timers
+
+```bash
+ssh -t ana-ml2 '
+  sudo resticprofile --config /etc/restic/profiles.yaml schedule --all &&
+  systemctl list-timers "resticprofile*"
+'
+```
+
+`resticprofile schedule` writes the unit files into `/etc/systemd/system/`
+and enables them. You should see three timers: backup, forget, check.
+
+### 6. Fire a manual backup to verify
+
+```bash
+ssh -t ana-ml2 'sudo resticprofile --config /etc/restic/profiles.yaml backup --verbose'
+```
+
+Expected output: a fresh snapshot id and a line summarizing added data
+size + file count.
+
+Cross-check from the hub:
+
+```bash
+ssh ana-docker 'docker exec backrest ls /repos/ana-ml2/'
+```
+
+Or in Backrest UI (http://10.250.50.70:9898), `ana-ml2` shows up as a new
+repo with one snapshot.
+
+## Ongoing
+
+Timers run at 01:00 / 03:00 / Sunday 05:00 (matching the rest of the
+fleet). Monitor via the `schedule-log` entries or Backrest's web UI.
+
+## Restoring
+
+```bash
+# List snapshots
+ssh -t ana-ml2 'sudo resticprofile --config /etc/restic/profiles.yaml snapshots'
+
+# Restore a single path
+ssh -t ana-ml2 'sudo resticprofile --config /etc/restic/profiles.yaml restore --target /tmp/restore latest --path /opt/docker'
+```
+
+## Gotchas specific to this host
+
+- **Large `/tank` is deliberately excluded** — don't edit source paths
+  to include it without a plan for the ~TB of model data.
+- **Repo passphrase is irreplaceable.** Losing `/etc/restic/password`
+  without a copy elsewhere = losing every snapshot in the `ana-ml2`
+  repo. Store a copy in your password manager on day one.
+- **`/var/lib/docker/volumes` assumes docker uses the default data
+  root.** If you ever switch to a custom dockerd data-root, update the
+  source list.

configs/restic/ana-ml2/profiles.yaml

@@ -0,0 +1,82 @@
+# resticprofile config for ana-ml2.
+#
+# **Bare metal** — not protected by any Proxmox vzdump. This is the only
+# backup for this host's config/state, so it's the highest-stakes repo in
+# the fleet. Writes to the Anaheim rest-server at 10.250.50.70 as user
+# `ana-ml2`. The full REST URL (with HTTP basic auth creds) lives in
+# /etc/restic/restic.env. The client-side encryption passphrase lives in
+# /etc/restic/password.
+#
+# Notable exclusions:
+#   - /tank/* is NOT in source paths. That's ~TB of model weights (HF
+#     caches, llama.cpp GGUFs, ComfyUI models, etc.) — all regenerable
+#     from upstream. Backing them up would blow the repo size budget.
+#   - No DB dumps needed. None of the stacks on this host (llama-swap,
+#     vllm-qwen3, comfyui, kokoro, parakeet, vibevoice, beszel-agent,
+#     dozzle-agent, dockge) store relational data.
+
+version: "1"
+
+global:
+  priority: low
+  ionice: true
+  ionice-class: 2
+  ionice-level: 7
+  min-memory: 100
+
+default:
+  env-file: /etc/restic/restic.env          # RESTIC_REPOSITORY=rest:http://user:pw@…
+  env:
+    RESTIC_PASSWORD_FILE: /etc/restic/password
+  initialize: false                         # repo created manually by `restic init`
+  lock: /var/lock/restic-ana-ml2.lock
+
+  backup:
+    verbose: 1
+    run-after:
+      - date +%s > /var/lib/restic/last-success
+    source:
+      - /opt/docker          # compose files + host-level configs — small (~100MB)
+      - /etc                 # host config (systemd units, chrony, apparmor, ssh, etc.)
+      - /root                # root shell history, ssh keys, any ad-hoc scripts
+      - /var/lib/docker/volumes  # named volumes (small; models live on /tank, not here)
+    exclude:
+      # Docker internals we never want in a backup
+      - /var/lib/docker/volumes/backingFsBlockDev
+      - /var/lib/docker/volumes/metadata.db
+      # Ephemeral / regenerable junk
+      - /opt/docker/compose/*/logs
+      - "**/*.log"
+      - "**/*.log.*"
+      - "**/*.pid"
+      # Root's noise — shell caches, tmp
+      - /root/.cache
+      - /root/.local/share/Trash
+      - /root/.npm
+      - /root/.python_history
+    tag:
+      - host:ana-ml2
+      - site:ana
+      - fleet:pfi
+    schedule: "*-*-* 01:00:00"
+    schedule-permission: system
+    schedule-log: /var/log/restic-backup.log
+
+  forget:
+    keep-daily: 7
+    keep-weekly: 4
+    keep-monthly: 12
+    keep-yearly: 3
+    # NOTE: no `prune: true` — rest-server runs with --append-only.
+    # See README.md "Prune ceremony".
+    tag:
+      - host:ana-ml2
+    schedule: "*-*-* 03:00:00"
+    schedule-permission: system
+    schedule-log: /var/log/restic-forget.log
+
+  check:
+    read-data-subset: 10%
+    schedule: "Sun *-*-* 05:00:00"
+    schedule-permission: system
+    schedule-log: /var/log/restic-check.log