esh-pfi-infrastructure/configs/restic/ana-ml2/README.md

restic / ana-ml2

Bare metal — ana-ml2 runs directly on a server chassis, not on any Proxmox hypervisor, so vzdump doesn't cover it. This is the sole backup for the host's configuration and is the highest-stakes backup target in the fleet.

What's backed up

Path	Purpose
/opt/docker	Compose files + config bind mounts (~110 MB)
/etc	Host config — systemd, ssh, chrony, apparmor, apt, etc.
/root	Root's ad-hoc scripts, shell history, ssh keys
/var/lib/docker/volumes	Named volumes used by running stacks (small — models live elsewhere)

What's not backed up (by design)

• /tank/* — model caches (HF hub, GGUFs, ComfyUI checkpoints, Kokoro voices, Vibevoice weights, Parakeet weights). Regenerable from upstream; storing them would blow the repo size budget.
• /root/.cache, /root/.npm, shell-noise caches
• Any *.log, *.pid, .../logs/ directories

Stacks on this host

All inference/AI-adjacent — none use relational databases, so no pre-backup DB hook is required. Contrast with configs/restic/ana-docker/ where synapse/seafile/vaultwarden DB dumps run first.

• llama-swap — GGUF swapper (llama.cpp)
• vllm-qwen3 — embedding + rerank
• comfyui, kokoro, parakeet, vibevoice
• beszel-agent-ana, dozzle-agent-ana, dockge

Deploy (one-time setup)

1. Credentials (repo already exists — reuse)

The Anaheim rest-server already has an ana-ml2 entry in its .htpasswd and a repo at /ana-ml2/ from a prior pass. Reusing both keeps the snapshot history consolidated.

On ana-ml2, install the two secret files from the existing values (both live in your password manager):

ssh -t ana-ml2 '
  sudo install -d -o root -g root -m 0700 /etc/restic /var/lib/restic &&
  sudo tee /etc/restic/restic.env > /dev/null <<EOF
RESTIC_REPOSITORY=rest:http://ana-ml2:<EXISTING-HTPASSWD-PASSWORD>@10.250.50.70:8000/ana-ml2/
EOF
  sudo chmod 600 /etc/restic/restic.env &&

  sudo tee /etc/restic/password > /dev/null <<EOF
<EXISTING-REPO-PASSPHRASE>
EOF
  sudo chmod 600 /etc/restic/password
'

2. Verify creds against the existing repo

Before deploying the profile, prove the secrets are correct. The /etc/restic/restic.env and /etc/restic/password files are root:600, so the env-file read AND the restic invocation both need to run inside the same sudo shell — otherwise $(cat …) runs as the login user, fails silently, and restic complains about a missing repository:

ssh -t ana-ml2 '
  sudo bash -c "
    set -a
    . /etc/restic/restic.env
    set +a
    RESTIC_PASSWORD_FILE=/etc/restic/password restic snapshots
  "
'

Expected: either a list of prior snapshots (from the earlier docker-files pass) or no snapshots found — both mean auth is good. If you see Fatal: wrong password or no key found, the passphrase in /etc/restic/password doesn't match the repo — check your password manager or rotate (see "Recreating the repo" below).

Do not run restic init — the repo is already initialized; init against an existing repo errors out deliberately.

If no repo exists yet (fresh setup)

Only applicable if you're starting from zero — otherwise skip.

ssh -t ana-docker '
  sudo htpasswd -B /path/to/rest-server/data/.htpasswd ana-ml2 &&  # prompts for password
  docker restart rest-server
'

ssh -t ana-ml2 '
  # Write /etc/restic/restic.env and /etc/restic/password as above, using
  # the fresh htpasswd password and a freshly generated passphrase:
  openssl rand -base64 48  # save this in your password manager IMMEDIATELY

  sudo env $(cat /etc/restic/restic.env) \
       RESTIC_PASSWORD_FILE=/etc/restic/password \
       restic init
'

3. Install resticprofile on ana-ml2

The host has restic but not resticprofile. From the resticprofile docs:

ssh -t ana-ml2 '
  curl -sfL https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh | sudo sh -s -- -b /usr/local/bin
  /usr/local/bin/resticprofile --version
'

4. Deploy the profile

scp configs/restic/ana-ml2/profiles.yaml ana-ml2:/tmp/profiles.yaml
ssh -t ana-ml2 '
  sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml &&
  rm /tmp/profiles.yaml
'

5. Generate + enable the systemd timers

ssh -t ana-ml2 '
  sudo resticprofile --config /etc/restic/profiles.yaml schedule --all &&
  systemctl list-timers "resticprofile*"
'

resticprofile schedule writes the unit files into /etc/systemd/system/ and enables them. You should see three timers: backup, forget, check.

6. Fire a manual backup to verify

ssh -t ana-ml2 'sudo resticprofile --config /etc/restic/profiles.yaml backup --verbose'

Expected output: a fresh snapshot id and a line summarizing added data size + file count.

Cross-check from the hub:

ssh ana-docker 'docker exec backrest ls /repos/ana-ml2/'

Or in Backrest UI (http://10.250.50.70:9898), ana-ml2 shows up as a new repo with one snapshot.

Ongoing

Timers run at 01:00 / 03:00 / Sunday 05:00 (matching the rest of the fleet). Monitor via the schedule-log entries or Backrest's web UI.

Restoring

# List snapshots
ssh -t ana-ml2 'sudo resticprofile --config /etc/restic/profiles.yaml snapshots'

# Restore a single path
ssh -t ana-ml2 'sudo resticprofile --config /etc/restic/profiles.yaml restore --target /tmp/restore latest --path /opt/docker'

Gotchas specific to this host

• Large /tank is deliberately excluded — don't edit source paths to include it without a plan for the ~TB of model data.
• Repo passphrase is irreplaceable. Losing /etc/restic/password without a copy elsewhere = losing every snapshot in the ana-ml2 repo. Store a copy in your password manager on day one.
• /var/lib/docker/volumes assumes docker uses the default data root. If you ever switch to a custom dockerd data-root, update the source list.