esh-pfi-infrastructure/configs/restic/irv-ml1/README.md

# restic / irv-ml1

**Bare metal, remote site** — irv-ml1 runs directly on a server chassis
at Irvine, reachable only via WireGuard tunnel from NH3. No vzdump
coverage. Analog of `ana-ml2`, but targets `rest-server-nh3` (not
`rest-server-ana`) because NH3 is the WG termination endpoint — keeps
the backup traffic on the same side of the tunnel.

## What's backed up

| Path | Purpose |
|---|---|
| `/opt/docker` | Compose files + config bind mounts |
| `/etc` | Host config — systemd, ssh, chrony, apparmor, **wg configs**, apt |
| `/root` | Root's ad-hoc scripts, shell history, ssh keys |
| `/home/lkraven` | Admin home — dotfiles, dev scripts, small configs |
| `/var/lib/docker/volumes` | Named volumes from dockerized stacks |
| `/worktank/comfyui/basedir/user` | ComfyUI workflows + per-user settings |
| `/worktank/comfyui/basedir/custom_nodes` | Hand-installed ComfyUI extensions |
| `/worktank/comfyui/basedir/input` | User-uploaded source images for workflows |
| `/worktank/cosyvoice/voices` | Cloned speaker profiles — reproducing them needs the original reference audio |

## What's **not** backed up (by design)

- **`/storetank/*`** — ~1.8 TB model weights. Regenerable from
  HuggingFace Hub / upstream repos. Do NOT add to source without a
  plan for the ~TB of data.
- **`/worktank/*` except the selective dirs above** — the rest is
  bulk, regenerable, or disposable:
    - `/worktank/comfyui/basedir/{models,output,temp}` — bulk weights
      + generated outputs + internal scratch
    - `/worktank/comfyui/run` — ~8 GB disposable Python venv + pip
      cache (container runtime for comfyui)
    - `/worktank/parakeet/models` — HF cache, re-downloads
    - `/worktank/cosyvoice/{input,output}` — scratch audio
- **Native AI stacks under `/opt/{ComfyUI,sillytavern,llama.cpp,...}`**
  — git clones + pip envs. Regenerable. Only `/opt/docker` (fleet
  convention for containerized stacks) is in the source list.
- **`/home/llmuser`** and **`/home/sduser`** — belong on `/storetank`
  / regenerable; kept out via explicit exclude.
- Home-dir `.cache/`, `.npm`, HF cache, conda envs, virtualenvs,
  `.cargo/registry`, Rust toolchain, `*.log`, `*.pid`
- Docker internals (`backingFsBlockDev`, `metadata.db`)

## Deploy (one-time setup)

### 1. Create rest-server-nh3 credentials for irv-ml1

On `nh3-nas` (where rest-server-nh3's data dir lives — `/volume1/Backup/restic/`),
add an htpasswd entry for user `irv-ml1`:

```bash
# Pick a strong password, store in password manager BEFORE running this
NEW_PW='<generated-strong-password>'

ssh -t nh3-nas '
  # rest-server-nh3 .htpasswd path under /volume1/Backup/restic/
  HTPATH=/volume1/Backup/restic/.htpasswd
  sudo docker run --rm httpd:2.4-alpine htpasswd -nbB irv-ml1 "'"$NEW_PW"'" \
    | sudo tee -a "$HTPATH" > /dev/null
  sudo tail -2 "$HTPATH"
'
# Restart rest-server-nh3 so it picks up the new credential
ssh -t nh3-nas 'sudo docker restart rest-server-nh3'
```

### 2. Install secret files on irv-ml1

```bash
ssh -t irv-ml1 '
  sudo install -d -o root -g root -m 0700 /etc/restic /var/lib/restic &&
  sudo tee /etc/restic/restic.env > /dev/null <<EOF
RESTIC_REPOSITORY=rest:http://irv-ml1:<htpasswd-password>@10.100.50.50:8000/irv-ml1/
EOF
  sudo chmod 600 /etc/restic/restic.env
'
```

Then install the repo passphrase (generate a fresh one — this is a
new repo, no prior passphrase to match):

```bash
ssh -t irv-ml1 'sudo bash -c "
  openssl rand -base64 48 | tr -d \"\\n\" > /etc/restic/password
  chmod 600 /etc/restic/password
  cat /etc/restic/password  # copy this to your password manager NOW
  echo
"'
```

**Copy the printed passphrase to your password manager before doing
anything else.** It prints exactly once. Lose it = lose the repo.

### 3. Initialize the repo

```bash
ssh -t irv-ml1 '
  sudo bash -c "
    set -a; . /etc/restic/restic.env; set +a
    RESTIC_PASSWORD_FILE=/etc/restic/password restic init
  "
'
```

Expected: `created restic repository <id> at rest:http://...`

### 4. Install resticprofile on irv-ml1

```bash
ssh -t irv-ml1 '
  curl -sfL https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh | sudo sh -s -- -b /usr/local/bin
  /usr/local/bin/resticprofile --version
'
```

### 5. Deploy the profile

```bash
scp configs/restic/irv-ml1/profiles.yaml irv-ml1:/tmp/profiles.yaml
ssh -t irv-ml1 '
  sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml &&
  rm /tmp/profiles.yaml
'
```

### 6. Generate + enable the systemd timers

```bash
ssh -t irv-ml1 '
  sudo resticprofile --config /etc/restic/profiles.yaml schedule --all &&
  systemctl list-timers "resticprofile*"
'
```

### 7. First manual backup

```bash
ssh -t irv-ml1 'sudo resticprofile --config /etc/restic/profiles.yaml backup --verbose'
```

Expect the first run to be a few hundred MB (host config + home dirs;
nothing from /storetank or /opt native stacks).

## WireGuard-specific gotcha

The backup traverses the WG tunnel. If the tunnel drops, the timer
will fail — restic doesn't auto-retry a missed window, and the timer
won't catch up until the next `01:00:00` fires. Consider:

- Monitor WG health separately (Beszel agent ping of 10.100.50.50)
- Expect more "failed backup" noise if WG is flaky; investigate
  the tunnel, not restic

## Ongoing

Timers fire at 01:00 / Sunday 05:00, matching the rest of the fleet.

## Restoring

Same pattern as ana-ml2 — see `configs/restic/ana-ml2/README.md` for
the full flow.

## Gotchas specific to this host

- **`/storetank` excluded entirely; `/worktank` included selectively.**
  Stacks that place user state under `/worktank/<stack>/` (comfyui,
  parakeet, cosyvoice) are cherry-picked by subtree; bulk/regenerable
  dirs stay out. See "What's backed up / not backed up" above.
- **Native AI stacks under `/opt/`** (ComfyUI, SillyTavern, etc.) are
  also NOT backed up. If the user fine-tunes config/weights inside
  those trees and wants to preserve specific paths, add them to the
  source list explicitly. (The dockerized ComfyUI at
  `/opt/docker/compose/comfyui` **is** covered via `/opt/docker`; only
  the native `/opt/ComfyUI` install is skipped.)
- **Backup traffic crosses the WG tunnel.** Latency is higher than
  in-LAN restic clients; first full backup may take longer than an
  equivalent ana-side host.
- **Repo passphrase is irreplaceable.** Losing `/etc/restic/password`
  without a copy elsewhere = losing every snapshot. Password manager
  on day one.