restic: ana-ml2 profile — cover the only bare-metal host in the fleet
ana-ml2 is not on any Proxmox hypervisor, so vzdump doesn't touch it. This closes the biggest single backup gap per the 2026-04-20 pipeline audit. Sources: /opt/docker (~110 MB), /etc, /root, /var/lib/docker/volumes. Excludes /tank/* (model weights — regenerable from Hugging Face and would blow repo size budget). No pre-backup DB hook — none of the llama-swap / vllm / comfyui / kokoro / parakeet / vibevoice stacks use relational databases. README walks through the one-time setup: rest-server .htpasswd entry, restic init with fresh passphrase, resticprofile install, systemd timer generation, verification against the Backrest UI.
This commit is contained in:
@@ -0,0 +1,151 @@
|
||||
# restic / ana-ml2
|
||||
|
||||
**Bare metal** — ana-ml2 runs directly on a server chassis, not on any
|
||||
Proxmox hypervisor, so vzdump doesn't cover it. This is the sole backup
|
||||
for the host's configuration and is the highest-stakes backup target in
|
||||
the fleet.
|
||||
|
||||
## What's backed up
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `/opt/docker` | Compose files + config bind mounts (~110 MB) |
|
||||
| `/etc` | Host config — systemd, ssh, chrony, apparmor, apt, etc. |
|
||||
| `/root` | Root's ad-hoc scripts, shell history, ssh keys |
|
||||
| `/var/lib/docker/volumes` | Named volumes used by running stacks (small — models live elsewhere) |
|
||||
|
||||
## What's **not** backed up (by design)
|
||||
|
||||
- **`/tank/*`** — model caches (HF hub, GGUFs, ComfyUI checkpoints,
|
||||
Kokoro voices, Vibevoice weights, Parakeet weights). Regenerable from
|
||||
upstream; storing them would blow the repo size budget.
|
||||
- `/root/.cache`, `/root/.npm`, shell-noise caches
|
||||
- Any `*.log`, `*.pid`, `.../logs/` directories
|
||||
|
||||
## Stacks on this host
|
||||
|
||||
All inference/AI-adjacent — none use relational databases, so no
|
||||
pre-backup DB hook is required. Contrast with `configs/restic/ana-docker/`
|
||||
where synapse/seafile/vaultwarden DB dumps run first.
|
||||
|
||||
- `llama-swap` — GGUF swapper (llama.cpp)
|
||||
- `vllm-qwen3` — embedding + rerank
|
||||
- `comfyui`, `kokoro`, `parakeet`, `vibevoice`
|
||||
- `beszel-agent-ana`, `dozzle-agent-ana`, `dockge`
|
||||
|
||||
## Deploy (one-time setup)
|
||||
|
||||
### 1. Add the client entry to the Anaheim rest-server .htpasswd
|
||||
|
||||
On ana-docker (where rest-server-ana runs):
|
||||
|
||||
```bash
|
||||
ssh -t ana-docker '
|
||||
cd /opt/docker/data/rest-server-ana && # wherever the data dir is
|
||||
sudo htpasswd -B /data/.htpasswd ana-ml2 && # prompts for a new password
|
||||
docker restart rest-server
|
||||
'
|
||||
```
|
||||
|
||||
Store the password where ana-ml2 can grab it. This is the **HTTP basic
|
||||
auth password**, not the restic repo passphrase — two separate secrets.
|
||||
|
||||
### 2. Create the restic repo
|
||||
|
||||
On ana-ml2, with a freshly generated encryption passphrase:
|
||||
|
||||
```bash
|
||||
ssh -t ana-ml2 '
|
||||
sudo install -d -o root -g root -m 0700 /etc/restic /var/lib/restic &&
|
||||
echo "RESTIC_REPOSITORY=rest:http://ana-ml2:<htpasswd-password>@10.250.50.70:8000/ana-ml2/" | sudo tee /etc/restic/restic.env > /dev/null &&
|
||||
sudo chmod 600 /etc/restic/restic.env &&
|
||||
|
||||
# Generate a fresh passphrase (print it to stdout exactly once — store
|
||||
# it in your password manager immediately; losing it = losing this
|
||||
# host´s entire backup history).
|
||||
openssl rand -base64 48 | sudo tee /etc/restic/password > /dev/null &&
|
||||
sudo chmod 600 /etc/restic/password &&
|
||||
|
||||
sudo env $(cat /etc/restic/restic.env) \
|
||||
RESTIC_PASSWORD_FILE=/etc/restic/password \
|
||||
restic init
|
||||
'
|
||||
```
|
||||
|
||||
### 3. Install resticprofile on ana-ml2
|
||||
|
||||
The host has `restic` but not `resticprofile`. From the resticprofile
|
||||
docs:
|
||||
|
||||
```bash
|
||||
ssh -t ana-ml2 '
|
||||
curl -sfL https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh | sudo sh -s -- -b /usr/local/bin
|
||||
/usr/local/bin/resticprofile --version
|
||||
'
|
||||
```
|
||||
|
||||
### 4. Deploy the profile
|
||||
|
||||
```bash
|
||||
scp configs/restic/ana-ml2/profiles.yaml ana-ml2:/tmp/profiles.yaml
|
||||
ssh -t ana-ml2 '
|
||||
sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml &&
|
||||
rm /tmp/profiles.yaml
|
||||
'
|
||||
```
|
||||
|
||||
### 5. Generate + enable the systemd timers
|
||||
|
||||
```bash
|
||||
ssh -t ana-ml2 '
|
||||
sudo resticprofile --config /etc/restic/profiles.yaml schedule --all &&
|
||||
systemctl list-timers "resticprofile*"
|
||||
'
|
||||
```
|
||||
|
||||
`resticprofile schedule` writes the unit files into `/etc/systemd/system/`
|
||||
and enables them. You should see three timers: backup, forget, check.
|
||||
|
||||
### 6. Fire a manual backup to verify
|
||||
|
||||
```bash
|
||||
ssh -t ana-ml2 'sudo resticprofile --config /etc/restic/profiles.yaml backup --verbose'
|
||||
```
|
||||
|
||||
Expected output: a fresh snapshot id and a line summarizing added data
|
||||
size + file count.
|
||||
|
||||
Cross-check from the hub:
|
||||
|
||||
```bash
|
||||
ssh ana-docker 'docker exec backrest ls /repos/ana-ml2/'
|
||||
```
|
||||
|
||||
Or in Backrest UI (http://10.250.50.70:9898), `ana-ml2` shows up as a new
|
||||
repo with one snapshot.
|
||||
|
||||
## Ongoing
|
||||
|
||||
Timers run at 01:00 / 03:00 / Sunday 05:00 (matching the rest of the
|
||||
fleet). Monitor via the `schedule-log` entries or Backrest's web UI.
|
||||
|
||||
## Restoring
|
||||
|
||||
```bash
|
||||
# List snapshots
|
||||
ssh -t ana-ml2 'sudo resticprofile --config /etc/restic/profiles.yaml snapshots'
|
||||
|
||||
# Restore a single path
|
||||
ssh -t ana-ml2 'sudo resticprofile --config /etc/restic/profiles.yaml restore --target /tmp/restore latest --path /opt/docker'
|
||||
```
|
||||
|
||||
## Gotchas specific to this host
|
||||
|
||||
- **Large `/tank` is deliberately excluded** — don't edit source paths
|
||||
to include it without a plan for the ~TB of model data.
|
||||
- **Repo passphrase is irreplaceable.** Losing `/etc/restic/password`
|
||||
without a copy elsewhere = losing every snapshot in the `ana-ml2`
|
||||
repo. Store a copy in your password manager on day one.
|
||||
- **`/var/lib/docker/volumes` assumes docker uses the default data
|
||||
root.** If you ever switch to a custom dockerd data-root, update the
|
||||
source list.
|
||||
Reference in New Issue
Block a user