esh-pfi-infrastructure/configs/restic/ana-ml2/README.md

# restic / ana-ml2

**Bare metal** — ana-ml2 runs directly on a server chassis, not on any
Proxmox hypervisor, so vzdump doesn't cover it. This is the sole backup
for the host's configuration and is the highest-stakes backup target in
the fleet.

## What's backed up

| Path | Purpose |
|---|---|
| `/opt/docker` | Compose files + config bind mounts (~110 MB) |
| `/etc` | Host config — systemd, ssh, chrony, apparmor, apt, etc. |
| `/root` | Root's ad-hoc scripts, shell history, ssh keys |
| `/var/lib/docker/volumes` | Named volumes used by running stacks (small — models live elsewhere) |

## What's **not** backed up (by design)

- **`/tank/*`** — model caches (HF hub, GGUFs, ComfyUI checkpoints,
  Kokoro voices, Vibevoice weights, Parakeet weights). Regenerable from
  upstream; storing them would blow the repo size budget.
- `/root/.cache`, `/root/.npm`, shell-noise caches
- Any `*.log`, `*.pid`, `.../logs/` directories

## Stacks on this host

All inference/AI-adjacent — none use relational databases, so no
pre-backup DB hook is required. Contrast with `configs/restic/ana-docker/`
where synapse/seafile/vaultwarden DB dumps run first.

- `llama-swap` — GGUF swapper (llama.cpp)
- `vllm` — embedding + rerank + reward classifier
- `comfyui`, `kokoro`, `parakeet`, `vibevoice`
- `beszel-agent-ana`, `dozzle-agent-ana`, `dockge`

## Deploy (one-time setup)

### 1. Credentials (repo already exists — reuse)

The Anaheim rest-server already has an `ana-ml2` entry in its `.htpasswd`
and a repo at `/ana-ml2/` from a prior pass. Reusing both keeps the
snapshot history consolidated.

On ana-ml2, install the two secret files from the existing values (both
live in your password manager):

```bash
ssh -t ana-ml2 '
  sudo install -d -o root -g root -m 0700 /etc/restic /var/lib/restic &&
  sudo tee /etc/restic/restic.env > /dev/null <<EOF
RESTIC_REPOSITORY=rest:http://ana-ml2:<EXISTING-HTPASSWD-PASSWORD>@10.250.50.70:8000/ana-ml2/
EOF
  sudo chmod 600 /etc/restic/restic.env &&

  sudo tee /etc/restic/password > /dev/null <<EOF
<EXISTING-REPO-PASSPHRASE>
EOF
  sudo chmod 600 /etc/restic/password
'
```

### 2. Verify creds against the existing repo

Before deploying the profile, prove the secrets are correct. The
`/etc/restic/restic.env` and `/etc/restic/password` files are root:600,
so the env-file read AND the restic invocation both need to run inside
the same sudo shell — otherwise `$(cat …)` runs as the login user,
fails silently, and restic complains about a missing repository:

```bash
ssh -t ana-ml2 '
  sudo bash -c "
    set -a
    . /etc/restic/restic.env
    set +a
    RESTIC_PASSWORD_FILE=/etc/restic/password restic snapshots
  "
'
```

Expected: either a list of prior snapshots (from the earlier
docker-files pass) or `no snapshots found` — both mean auth is good.
If you see `Fatal: wrong password or no key found`, the passphrase in
`/etc/restic/password` doesn't match the repo — check your password
manager or rotate (see "Recreating the repo" below).

**Do not run `restic init`** — the repo is already initialized; init
against an existing repo errors out deliberately.

### Recreating the repo (lost passphrase / starting fresh)

If `restic snapshots` returns `wrong password or no key found` and the
passphrase can't be recovered, the old repo's data is unrecoverable and
the cleanest path is to wipe and reinit. If you're truly starting from
zero (no prior htpasswd entry or repo at all), skip the
`htpasswd` step — only the init is needed.

The rest-server's data root is `/mnt/backup/restic/repo/ana/` on
ana-docker (NFS-mounted from the Debian NAS at 10.250.50.50). Per-host
repos are direct subdirs — for `ana-ml2` that's
`/mnt/backup/restic/repo/ana/ana-ml2/`. See
`stacks/rest-server-ana/README.md` for the full sibling layout
(`ana/`, `esh/`, `nh3/`).

```bash
# 1. Wipe the old repo content on ana-docker. `--append-only` blocks
#    restic's own delete ops but not direct filesystem removal — this
#    is intentional for exactly this case.
ssh -t ana-docker '
  sudo rm -rf /mnt/backup/restic/repo/ana/ana-ml2 &&
  sudo docker restart rest-server
'

# 2. Optional: rotate the htpasswd password for ana-ml2 at the same time.
#    Produces a new hash; replace in place. (Skip if the existing
#    htpasswd password is still trusted.)
ssh ana-docker 'docker run --rm httpd:2.4-alpine htpasswd -nbB ana-ml2 "<new-pw>"' \
  | ssh -t ana-docker 'sudo sed -i "/^ana-ml2:/d" /mnt/backup/restic/repo/ana/.htpasswd && sudo tee -a /mnt/backup/restic/repo/ana/.htpasswd >/dev/null'

# 3a. Init the repo — interactively prompt for a passphrase you generated
#     in your password manager ahead of time. restic asks twice (init +
#     confirm). This keeps the passphrase out of shell history / transcripts.
ssh -t ana-ml2 '
  sudo bash -c "
    set -a
    . /etc/restic/restic.env
    set +a
    restic init
  "
'

# 3b. Install the same passphrase into /etc/restic/password so timers can
#     run unattended. `cat > file` + Ctrl-D avoids the passphrase ever
#     landing in your shell history or the ssh command line.
ssh -t ana-ml2 'sudo bash -c "cat > /etc/restic/password && chmod 600 /etc/restic/password"'
#   Terminal waits for input:
#   - paste the same passphrase
#   - press Enter
#   - press Ctrl-D on an empty line
#   (restic strips the trailing newline when reading the file.)

# 3c. Verify the file-based passphrase agrees with what init registered.
ssh -t ana-ml2 '
  sudo bash -c "
    set -a
    . /etc/restic/restic.env
    set +a
    RESTIC_PASSWORD_FILE=/etc/restic/password restic snapshots
  "
'
#   Expect: "no snapshots found" — repo is initialized and both paths
#   (interactive and file-based) decrypt it.
```

**Alternative (random machine-generated passphrase):** if you want restic
to generate the passphrase rather than supplying one from your password
manager, replace 3a/3b with:

```bash
ssh -t ana-ml2 '
  sudo bash -c "
    openssl rand -base64 48 | tr -d \"\\n\" > /etc/restic/password
    chmod 600 /etc/restic/password
    cat /etc/restic/password  # copy this to your password manager NOW
    echo
    set -a; . /etc/restic/restic.env; set +a
    RESTIC_PASSWORD_FILE=/etc/restic/password restic init
  "
'
```

The passphrase prints to the terminal exactly once — don't close the
window before copying it. Losing it = losing this repo.

### 3. Install resticprofile on ana-ml2

The host has `restic` but not `resticprofile`. From the resticprofile
docs:

```bash
ssh -t ana-ml2 '
  curl -sfL https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh | sudo sh -s -- -b /usr/local/bin
  /usr/local/bin/resticprofile --version
'
```

### 4. Deploy the profile

```bash
scp configs/restic/ana-ml2/profiles.yaml ana-ml2:/tmp/profiles.yaml
ssh -t ana-ml2 '
  sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml &&
  rm /tmp/profiles.yaml
'
```

### 5. Generate + enable the systemd timers

```bash
ssh -t ana-ml2 '
  sudo resticprofile --config /etc/restic/profiles.yaml schedule --all &&
  systemctl list-timers "resticprofile*"
'
```

`resticprofile schedule` writes the unit files into `/etc/systemd/system/`
and enables them. You should see three timers: backup, forget, check.

### 6. Fire a manual backup to verify

```bash
ssh -t ana-ml2 'sudo resticprofile --config /etc/restic/profiles.yaml backup --verbose'
```

Expected output: a fresh snapshot id and a line summarizing added data
size + file count.

Cross-check from the hub:

```bash
ssh ana-docker 'docker exec backrest ls /repos/ana-ml2/'
```

Or in Backrest UI (http://10.250.50.70:9898), `ana-ml2` shows up as a new
repo with one snapshot.

## Ongoing

Timers run at 01:00 / 03:00 / Sunday 05:00 (matching the rest of the
fleet). Monitor via the `schedule-log` entries or Backrest's web UI.

## Restoring

```bash
# List snapshots
ssh -t ana-ml2 'sudo resticprofile --config /etc/restic/profiles.yaml snapshots'

# Restore a single path
ssh -t ana-ml2 'sudo resticprofile --config /etc/restic/profiles.yaml restore --target /tmp/restore latest --path /opt/docker'
```

## Gotchas specific to this host

- **Large `/tank` is deliberately excluded** — don't edit source paths
  to include it without a plan for the ~TB of model data.
- **Repo passphrase is irreplaceable.** Losing `/etc/restic/password`
  without a copy elsewhere = losing every snapshot in the `ana-ml2`
  repo. Store a copy in your password manager on day one.
- **`/var/lib/docker/volumes` assumes docker uses the default data
  root.** If you ever switch to a custom dockerd data-root, update the
  source list.