esh-pfi-infrastructure/configs/restic/irv-ml1

restic / irv-ml1

Bare metal, remote site — irv-ml1 runs directly on a server chassis at Irvine, reachable only via WireGuard tunnel from NH3. No vzdump coverage. Analog of ana-ml2, but targets rest-server-nh3 (not rest-server-ana) because NH3 is the WG termination endpoint — keeps the backup traffic on the same side of the tunnel.

What's backed up

Path	Purpose
/opt/docker	Compose files + config bind mounts
/etc	Host config — systemd, ssh, chrony, apparmor, wg configs, apt
/root	Root's ad-hoc scripts, shell history, ssh keys
/home/lkraven	Admin home — dotfiles, dev scripts, small configs
/var/lib/docker/volumes	Named volumes from dockerized stacks
/worktank/comfyui/basedir/user	ComfyUI workflows + per-user settings
/worktank/comfyui/basedir/custom_nodes	Hand-installed ComfyUI extensions
/worktank/comfyui/basedir/input	User-uploaded source images for workflows
/worktank/cosyvoice/voices	Cloned speaker profiles — reproducing them needs the original reference audio

What's not backed up (by design)

• /storetank/* — ~1.8 TB model weights. Regenerable from HuggingFace Hub / upstream repos. Do NOT add to source without a plan for the ~TB of data.
• /worktank/* except the selective dirs above — the rest is bulk, regenerable, or disposable:
  • /worktank/comfyui/basedir/{models,output,temp} — bulk weights
    • generated outputs + internal scratch
  • /worktank/comfyui/run — ~8 GB disposable Python venv + pip cache (container runtime for comfyui)
  • /worktank/parakeet/models — HF cache, re-downloads
  • /worktank/cosyvoice/{input,output} — scratch audio
• Native AI stacks under /opt/{ComfyUI,sillytavern,llama.cpp,...} — git clones + pip envs. Regenerable. Only /opt/docker (fleet convention for containerized stacks) is in the source list.
• /home/llmuser and /home/sduser — belong on /storetank / regenerable; kept out via explicit exclude.
• Home-dir .cache/, .npm, HF cache, conda envs, virtualenvs, .cargo/registry, Rust toolchain, *.log, *.pid
• Docker internals (backingFsBlockDev, metadata.db)

Deploy (one-time setup)

1. Create rest-server-nh3 credentials for irv-ml1

On nh3-nas (where rest-server-nh3's data dir lives — /volume1/Backup/restic/), add an htpasswd entry for user irv-ml1:

# Pick a strong password, store in password manager BEFORE running this
NEW_PW='<generated-strong-password>'

ssh -t nh3-nas '
  # rest-server-nh3 .htpasswd path under /volume1/Backup/restic/
  HTPATH=/volume1/Backup/restic/.htpasswd
  sudo docker run --rm httpd:2.4-alpine htpasswd -nbB irv-ml1 "'"$NEW_PW"'" \
    | sudo tee -a "$HTPATH" > /dev/null
  sudo tail -2 "$HTPATH"
'
# Restart rest-server-nh3 so it picks up the new credential
ssh -t nh3-nas 'sudo docker restart rest-server-nh3'

2. Install secret files on irv-ml1

ssh -t irv-ml1 '
  sudo install -d -o root -g root -m 0700 /etc/restic /var/lib/restic &&
  sudo tee /etc/restic/restic.env > /dev/null <<EOF
RESTIC_REPOSITORY=rest:http://irv-ml1:<htpasswd-password>@10.100.50.50:8000/irv-ml1/
EOF
  sudo chmod 600 /etc/restic/restic.env
'

Then install the repo passphrase (generate a fresh one — this is a new repo, no prior passphrase to match):

ssh -t irv-ml1 'sudo bash -c "
  openssl rand -base64 48 | tr -d \"\\n\" > /etc/restic/password
  chmod 600 /etc/restic/password
  cat /etc/restic/password  # copy this to your password manager NOW
  echo
"'

Copy the printed passphrase to your password manager before doing anything else. It prints exactly once. Lose it = lose the repo.

3. Initialize the repo

ssh -t irv-ml1 '
  sudo bash -c "
    set -a; . /etc/restic/restic.env; set +a
    RESTIC_PASSWORD_FILE=/etc/restic/password restic init
  "
'

Expected: created restic repository <id> at rest:http://...

4. Install resticprofile on irv-ml1

ssh -t irv-ml1 '
  curl -sfL https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh | sudo sh -s -- -b /usr/local/bin
  /usr/local/bin/resticprofile --version
'

5. Deploy the profile

scp configs/restic/irv-ml1/profiles.yaml irv-ml1:/tmp/profiles.yaml
ssh -t irv-ml1 '
  sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml &&
  rm /tmp/profiles.yaml
'

6. Generate + enable the systemd timers

ssh -t irv-ml1 '
  sudo resticprofile --config /etc/restic/profiles.yaml schedule --all &&
  systemctl list-timers "resticprofile*"
'

7. First manual backup

ssh -t irv-ml1 'sudo resticprofile --config /etc/restic/profiles.yaml backup --verbose'

Expect the first run to be a few hundred MB (host config + home dirs; nothing from /storetank or /opt native stacks).

WireGuard-specific gotcha

The backup traverses the WG tunnel. If the tunnel drops, the timer will fail — restic doesn't auto-retry a missed window, and the timer won't catch up until the next 01:00:00 fires. Consider:

• Monitor WG health separately (Beszel agent ping of 10.100.50.50)
• Expect more "failed backup" noise if WG is flaky; investigate the tunnel, not restic

Ongoing

Timers fire at 01:00 / Sunday 05:00, matching the rest of the fleet.

Restoring

Same pattern as ana-ml2 — see configs/restic/ana-ml2/README.md for the full flow.

Gotchas specific to this host

• /storetank excluded entirely; /worktank included selectively. Stacks that place user state under /worktank/<stack>/ (comfyui, parakeet, cosyvoice) are cherry-picked by subtree; bulk/regenerable dirs stay out. See "What's backed up / not backed up" above.
• Native AI stacks under /opt/ (ComfyUI, SillyTavern, etc.) are also NOT backed up. If the user fine-tunes config/weights inside those trees and wants to preserve specific paths, add them to the source list explicitly. (The dockerized ComfyUI at /opt/docker/compose/comfyui is covered via /opt/docker; only the native /opt/ComfyUI install is skipped.)
• Backup traffic crosses the WG tunnel. Latency is higher than in-LAN restic clients; first full backup may take longer than an equivalent ana-side host.
• Repo passphrase is irreplaceable. Losing /etc/restic/password without a copy elsewhere = losing every snapshot. Password manager on day one.