esh-pfi-infrastructure/configs/restic/ana-ml2/README.md

restic / ana-ml2

Bare metal — ana-ml2 runs directly on a server chassis, not on any Proxmox hypervisor, so vzdump doesn't cover it. This is the sole backup for the host's configuration and is the highest-stakes backup target in the fleet.

What's backed up

Path	Purpose
/opt/docker	Compose files + config bind mounts (~110 MB)
/etc	Host config — systemd, ssh, chrony, apparmor, apt, etc.
/root	Root's ad-hoc scripts, shell history, ssh keys
/var/lib/docker/volumes	Named volumes used by running stacks (small — models live elsewhere)

What's not backed up (by design)

• /tank/* — model caches (HF hub, GGUFs, ComfyUI checkpoints, Kokoro voices, Vibevoice weights, Parakeet weights). Regenerable from upstream; storing them would blow the repo size budget.
• /root/.cache, /root/.npm, shell-noise caches
• Any *.log, *.pid, .../logs/ directories

Stacks on this host

All inference/AI-adjacent — none use relational databases, so no pre-backup DB hook is required. Contrast with configs/restic/ana-docker/ where synapse/seafile/vaultwarden DB dumps run first.

• llama-swap — GGUF swapper (llama.cpp)
• vllm-qwen3 — embedding + rerank
• comfyui, kokoro, parakeet, vibevoice
• beszel-agent-ana, dozzle-agent-ana, dockge

Deploy (one-time setup)

1. Add the client entry to the Anaheim rest-server .htpasswd

On ana-docker (where rest-server-ana runs):

ssh -t ana-docker '
  cd /opt/docker/data/rest-server-ana &&   # wherever the data dir is
  sudo htpasswd -B /data/.htpasswd ana-ml2 &&   # prompts for a new password
  docker restart rest-server
'

Store the password where ana-ml2 can grab it. This is the HTTP basic auth password, not the restic repo passphrase — two separate secrets.

2. Create the restic repo

On ana-ml2, with a freshly generated encryption passphrase:

ssh -t ana-ml2 '
  sudo install -d -o root -g root -m 0700 /etc/restic /var/lib/restic &&
  echo "RESTIC_REPOSITORY=rest:http://ana-ml2:<htpasswd-password>@10.250.50.70:8000/ana-ml2/" | sudo tee /etc/restic/restic.env > /dev/null &&
  sudo chmod 600 /etc/restic/restic.env &&

  # Generate a fresh passphrase (print it to stdout exactly once — store
  # it in your password manager immediately; losing it = losing this
  # host´s entire backup history).
  openssl rand -base64 48 | sudo tee /etc/restic/password > /dev/null &&
  sudo chmod 600 /etc/restic/password &&

  sudo env $(cat /etc/restic/restic.env) \
       RESTIC_PASSWORD_FILE=/etc/restic/password \
       restic init
'

3. Install resticprofile on ana-ml2

The host has restic but not resticprofile. From the resticprofile docs:

ssh -t ana-ml2 '
  curl -sfL https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh | sudo sh -s -- -b /usr/local/bin
  /usr/local/bin/resticprofile --version
'

4. Deploy the profile

scp configs/restic/ana-ml2/profiles.yaml ana-ml2:/tmp/profiles.yaml
ssh -t ana-ml2 '
  sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml &&
  rm /tmp/profiles.yaml
'

5. Generate + enable the systemd timers

ssh -t ana-ml2 '
  sudo resticprofile --config /etc/restic/profiles.yaml schedule --all &&
  systemctl list-timers "resticprofile*"
'

resticprofile schedule writes the unit files into /etc/systemd/system/ and enables them. You should see three timers: backup, forget, check.

6. Fire a manual backup to verify

ssh -t ana-ml2 'sudo resticprofile --config /etc/restic/profiles.yaml backup --verbose'

Expected output: a fresh snapshot id and a line summarizing added data size + file count.

Cross-check from the hub:

ssh ana-docker 'docker exec backrest ls /repos/ana-ml2/'

Or in Backrest UI (http://10.250.50.70:9898), ana-ml2 shows up as a new repo with one snapshot.

Ongoing

Timers run at 01:00 / 03:00 / Sunday 05:00 (matching the rest of the fleet). Monitor via the schedule-log entries or Backrest's web UI.

Restoring

# List snapshots
ssh -t ana-ml2 'sudo resticprofile --config /etc/restic/profiles.yaml snapshots'

# Restore a single path
ssh -t ana-ml2 'sudo resticprofile --config /etc/restic/profiles.yaml restore --target /tmp/restore latest --path /opt/docker'

Gotchas specific to this host

• Large /tank is deliberately excluded — don't edit source paths to include it without a plan for the ~TB of model data.
• Repo passphrase is irreplaceable. Losing /etc/restic/password without a copy elsewhere = losing every snapshot in the ana-ml2 repo. Store a copy in your password manager on day one.
• /var/lib/docker/volumes assumes docker uses the default data root. If you ever switch to a custom dockerd data-root, update the source list.