vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

restic/irv-ml1: cover docker-stack user state under /worktank

Browse Source 
Selectively include /worktank/<stack>/ subtrees now that comfyui,
parakeet, and cosyvoice place real user state there. Bulk weights,
scratch outputs, and the ~8 GB disposable comfyui run/ venv stay
out — both by the include list being precise and by belt-and-
suspenders exclude patterns.

Added sources:
  /worktank/comfyui/basedir/user            workflows + per-user settings
  /worktank/comfyui/basedir/custom_nodes    hand-installed extensions
  /worktank/comfyui/basedir/input           user-uploaded source images
  /worktank/cosyvoice/voices                cloned speaker profiles

Belt-and-suspenders excludes (inert under current sources; guards
against a future wholesale-add of /worktank):
  /worktank/comfyui/basedir/{models,output,temp}
  /worktank/comfyui/run
  /worktank/parakeet/models
  /worktank/cosyvoice/{input,output}

Verified by `resticprofile backup --dry-run` on irv-ml1 — 1.5 GiB
scanned across all 9 sources, 13 MiB new to the repo. If any bulk
dir had leaked in, the total would be multi-GB.

Also fixed a stale /home row in the README (profile only sources
/home/lkraven; llmuser + sduser are explicitly excluded).
This commit is contained in:
Vuong Hoang
2026-04-23 23:48:23 -07:00
parent 1a67370138
commit 82f95d7428
2 changed files with 318 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
configs/restic/irv-ml1/README.md
+180
View File
@@ -0,0 +1,180 @@
# restic / irv-ml1
**Bare metal, remote site** — irv-ml1 runs directly on a server chassis
at Irvine, reachable only via WireGuard tunnel from NH3. No vzdump
coverage. Analog of `ana-ml2`, but targets `rest-server-nh3` (not
`rest-server-ana`) because NH3 is the WG termination endpoint — keeps
the backup traffic on the same side of the tunnel.
## What's backed up
| Path | Purpose |
|---|---|
| `/opt/docker` | Compose files + config bind mounts |
| `/etc` | Host config — systemd, ssh, chrony, apparmor, **wg configs**, apt |
| `/root` | Root's ad-hoc scripts, shell history, ssh keys |
| `/home/lkraven` | Admin home — dotfiles, dev scripts, small configs |
| `/var/lib/docker/volumes` | Named volumes from dockerized stacks |
| `/worktank/comfyui/basedir/user` | ComfyUI workflows + per-user settings |
| `/worktank/comfyui/basedir/custom_nodes` | Hand-installed ComfyUI extensions |
| `/worktank/comfyui/basedir/input` | User-uploaded source images for workflows |
| `/worktank/cosyvoice/voices` | Cloned speaker profiles — reproducing them needs the original reference audio |
## What's **not** backed up (by design)
- **`/storetank/*`** — ~1.8 TB model weights. Regenerable from
HuggingFace Hub / upstream repos. Do NOT add to source without a
plan for the ~TB of data.
- **`/worktank/*` except the selective dirs above** — the rest is
bulk, regenerable, or disposable:
- `/worktank/comfyui/basedir/{models,output,temp}` — bulk weights
+ generated outputs + internal scratch
- `/worktank/comfyui/run` — ~8 GB disposable Python venv + pip
cache (container runtime for comfyui)
- `/worktank/parakeet/models` — HF cache, re-downloads
- `/worktank/cosyvoice/{input,output}` — scratch audio
- **Native AI stacks under `/opt/{ComfyUI,sillytavern,llama.cpp,...}`**
— git clones + pip envs. Regenerable. Only `/opt/docker` (fleet
convention for containerized stacks) is in the source list.
- **`/home/llmuser`** and **`/home/sduser`** — belong on `/storetank`
/ regenerable; kept out via explicit exclude.
- Home-dir `.cache/`, `.npm`, HF cache, conda envs, virtualenvs,
`.cargo/registry`, Rust toolchain, `*.log`, `*.pid`
- Docker internals (`backingFsBlockDev`, `metadata.db`)
## Deploy (one-time setup)
### 1. Create rest-server-nh3 credentials for irv-ml1
On `nh3-nas` (where rest-server-nh3's data dir lives — `/volume1/Backup/restic/`),
add an htpasswd entry for user `irv-ml1`:
```bash
# Pick a strong password, store in password manager BEFORE running this
NEW_PW='<generated-strong-password>'
ssh -t nh3-nas '
# rest-server-nh3 .htpasswd path under /volume1/Backup/restic/
HTPATH=/volume1/Backup/restic/.htpasswd
sudo docker run --rm httpd:2.4-alpine htpasswd -nbB irv-ml1 "'"$NEW_PW"'" \
| sudo tee -a "$HTPATH" > /dev/null
sudo tail -2 "$HTPATH"
'
# Restart rest-server-nh3 so it picks up the new credential
ssh -t nh3-nas 'sudo docker restart rest-server-nh3'
```
### 2. Install secret files on irv-ml1
```bash
ssh -t irv-ml1 '
sudo install -d -o root -g root -m 0700 /etc/restic /var/lib/restic &&
sudo tee /etc/restic/restic.env > /dev/null <<EOF
RESTIC_REPOSITORY=rest:http://irv-ml1:<htpasswd-password>@10.100.50.50:8000/irv-ml1/
EOF
sudo chmod 600 /etc/restic/restic.env
'
```
Then install the repo passphrase (generate a fresh one — this is a
new repo, no prior passphrase to match):
```bash
ssh -t irv-ml1 'sudo bash -c "
openssl rand -base64 48 | tr -d \"\\n\" > /etc/restic/password
chmod 600 /etc/restic/password
cat /etc/restic/password # copy this to your password manager NOW
echo
"'
```
**Copy the printed passphrase to your password manager before doing
anything else.** It prints exactly once. Lose it = lose the repo.
### 3. Initialize the repo
```bash
ssh -t irv-ml1 '
sudo bash -c "
set -a; . /etc/restic/restic.env; set +a
RESTIC_PASSWORD_FILE=/etc/restic/password restic init
"
'
```
Expected: `created restic repository <id> at rest:http://...`
### 4. Install resticprofile on irv-ml1
```bash
ssh -t irv-ml1 '
curl -sfL https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh | sudo sh -s -- -b /usr/local/bin
/usr/local/bin/resticprofile --version
'
```
### 5. Deploy the profile
```bash
scp configs/restic/irv-ml1/profiles.yaml irv-ml1:/tmp/profiles.yaml
ssh -t irv-ml1 '
sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml &&
rm /tmp/profiles.yaml
'
```
### 6. Generate + enable the systemd timers
```bash
ssh -t irv-ml1 '
sudo resticprofile --config /etc/restic/profiles.yaml schedule --all &&
systemctl list-timers "resticprofile*"
'
```
### 7. First manual backup
```bash
ssh -t irv-ml1 'sudo resticprofile --config /etc/restic/profiles.yaml backup --verbose'
```
Expect the first run to be a few hundred MB (host config + home dirs;
nothing from /storetank or /opt native stacks).
## WireGuard-specific gotcha
The backup traverses the WG tunnel. If the tunnel drops, the timer
will fail — restic doesn't auto-retry a missed window, and the timer
won't catch up until the next `01:00:00` fires. Consider:
- Monitor WG health separately (Beszel agent ping of 10.100.50.50)
- Expect more "failed backup" noise if WG is flaky; investigate
the tunnel, not restic
## Ongoing
Timers fire at 01:00 / Sunday 05:00, matching the rest of the fleet.
## Restoring
Same pattern as ana-ml2 — see `configs/restic/ana-ml2/README.md` for
the full flow.
## Gotchas specific to this host
- **`/storetank` excluded entirely; `/worktank` included selectively.**
Stacks that place user state under `/worktank/<stack>/` (comfyui,
parakeet, cosyvoice) are cherry-picked by subtree; bulk/regenerable
dirs stay out. See "What's backed up / not backed up" above.
- **Native AI stacks under `/opt/`** (ComfyUI, SillyTavern, etc.) are
also NOT backed up. If the user fine-tunes config/weights inside
those trees and wants to preserve specific paths, add them to the
source list explicitly. (The dockerized ComfyUI at
`/opt/docker/compose/comfyui` **is** covered via `/opt/docker`; only
the native `/opt/ComfyUI` install is skipped.)
- **Backup traffic crosses the WG tunnel.** Latency is higher than
in-LAN restic clients; first full backup may take longer than an
equivalent ana-side host.
- **Repo passphrase is irreplaceable.** Losing `/etc/restic/password`
without a copy elsewhere = losing every snapshot. Password manager
on day one.
configs/restic/irv-ml1/profiles.yaml
+138
View File
@@ -0,0 +1,138 @@
# resticprofile config for irv-ml1.
#
# **Bare metal, remote site.** Not protected by any Proxmox vzdump — this
# is the sole backup for this host. Reachable only via WireGuard
# (10.100.79.3 is the WG tunnel IP). Backups target rest-server-nh3 at
# 10.100.50.50 because that's the closest endpoint through the WG tunnel;
# cross-site to ana-docker would double-hop the WAN.
#
# Notable exclusions:
# - /storetank/* — ~1.8 TB of model weights (ComfyUI checkpoints, HF hub,
# llama.cpp GGUFs, etc.). All regenerable from upstream. Do NOT add.
# - /worktank/* — *selectively* included. Docker stacks on this host now
# place user state under /worktank/<stack>/ (comfyui, parakeet,
# cosyvoice). Sources are explicit per-subtree; bulk/scratch/run
# dirs stay out. Rule of thumb: precious human-generated data in,
# downloadable weights + regenerable outputs + disposable runtime
# out.
# - Native AI toolchain under /opt/{ComfyUI,sillytavern,stablediffusion,
# llama.cpp,kokoro,...} — git clones / pip envs, regenerable from
# source. Only /opt/docker is backed up (compose files + configs,
# once Docker is installed following the fleet convention).
#
# No DB dumps needed — this host's workloads don't store relational data.
version: "1"
global:
priority: low
ionice: true
ionice-class: 2
ionice-level: 7
min-memory: 100
default:
env-file: /etc/restic/restic.env # RESTIC_REPOSITORY=rest:http://user:pw@10.100.50.50:8000/irv-ml1/
env:
RESTIC_PASSWORD_FILE: /etc/restic/password
initialize: false # repo created manually by `restic init`
lock: /var/lock/restic-irv-ml1.lock
backup:
verbose: 1
run-after:
- date +%s > /var/lib/restic/last-success
source:
- /opt/docker # compose files + bind-mount configs
- /etc # host config — systemd, chrony, apparmor, ssh, wg
- /root # root's ad-hoc scripts, shell history, ssh keys
- /home/lkraven # primary admin home dir — dotfiles, dev scripts, small configs
- /var/lib/docker/volumes # named volumes from dockerized stacks
# Selective /worktank inclusions (the rest of /worktank stays out):
- /worktank/comfyui/basedir/user # ComfyUI workflows + settings
- /worktank/comfyui/basedir/custom_nodes # hand-installed ComfyUI extensions
- /worktank/comfyui/basedir/input # user-uploaded source images for workflows
- /worktank/cosyvoice/voices # cloned speaker profiles — reproducing them needs the original reference audio
# Deliberately NOT in source (see "What's not backed up" in README):
# /home/llmuser # ~380 GB of model weights / training outputs
# # (contents belong on /storetank; regenerable from HF)
# /home/sduser # Stable Diffusion user state (~2.6 GB); scope-check
# # before adding — likely regenerable
# /worktank/comfyui/basedir/{models,output,temp} # bulk + regenerable
# /worktank/comfyui/run # ~8 GB disposable venv
# /worktank/parakeet/models # HF cache, regenerable
# /worktank/cosyvoice/{input,output} # scratch
exclude:
# Explicit protection: even if source is ever changed to /home, skip
# llmuser entirely — the only reason it's 380 GB is AI artifacts that
# belong on /storetank or HF hub, not in a config-backup repo.
- /home/llmuser
- /home/sduser
# Defense-in-depth: even if `source` is ever broadened to /worktank
# wholesale, these must never enter the repo.
- /worktank/comfyui/basedir/models
- /worktank/comfyui/basedir/output
- /worktank/comfyui/basedir/temp
- /worktank/comfyui/run
- /worktank/parakeet/models
- /worktank/cosyvoice/input
- /worktank/cosyvoice/output
# Docker internals we never want in a backup
- /var/lib/docker/volumes/backingFsBlockDev
- /var/lib/docker/volumes/metadata.db
# Ephemeral / regenerable junk
- /opt/docker/compose/*/logs
- "**/*.log"
- "**/*.log.*"
- "**/*.pid"
# User-cache / toolchain noise under home dirs (saves ~5.8 GB on /home/lkraven alone)
- /home/*/.cache
- /home/*/.npm
- /home/*/.local/share/Trash
- /home/*/.cargo/registry
- /home/*/.cargo/git
- /home/*/.rustup
- /home/*/huggingface
- /home/*/.cache/huggingface
- /home/*/conda
- /home/*/anaconda3
- /home/*/miniconda3
- /home/*/.conda
- /home/*/micromamba
# Common Python virtualenv noise
- "**/__pycache__"
- "**/.venv"
- "**/venv"
- "**/env"
- "**/.pytest_cache"
# Root's noise
- /root/.cache
- /root/.local/share/Trash
- /root/.npm
- /root/.python_history
tag:
- host:irv-ml1
- site:irv
- fleet:pfi
schedule: "*-*-* 01:00:00"
schedule-permission: system
schedule-log: /var/log/restic-backup.log
forget:
keep-daily: 7
keep-weekly: 4
keep-monthly: 12
keep-yearly: 3
# NOTE: no `prune: true` — rest-server runs with --append-only.
# See README.md "Prune ceremony".
tag:
- host:irv-ml1
# Schedule removed at deploy: forget against --append-only rest-server
# always fails (delete ops blocked). Run manually during the prune
# ceremony when --append-only is temporarily off.
check:
read-data-subset: 10%
schedule: "Sun *-*-* 05:00:00"
schedule-permission: system
schedule-log: /var/log/restic-check.log