vh
d8f59a15d9
esh-ml1 is outside vzdump, so augaman's face gallery reaches backup only through restic. New playbooks/esh-ml1-restic.yaml installs restic 0.14.0 (the same Debian package as the other ESH hosts) and resticprofile 0.33.1 (pinned, sha256-checked). It uploads configs/restic/esh-ml1/ and schedules a daily 0100 PT backup plus a Sunday 0500 PT check to rest-server-ana. The CT runs UTC, so both schedules name the zone explicitly. pre-backup.sh is fail-closed: it runs augaman's own backup CLI, and any failure, including a stopped container, aborts the run. Tested with a stub docker that exits 1: the run returned 1, and neither the snapshot count nor last-success moved. The restore was verified at identity level against augaman-dev's public-domain canary (snapshot fd3061a1: the restored copy's digest over identities and samples matches the live gallery). That meets the operator gate for real enrollments. The repository URL is read through repository-file rather than restic.env. resticprofile schedule copies env-file values into world-readable systemd units, which publishes the rest-server password on the env-file hosts (observed on esh-docker-vm). This is recorded in the backups runbook under Known gaps, and the playbook verifies no generated unit contains the URL. esh-ml1 is added to the freshness check's expected ana-side repos and to the runbook tables. augaman moves to v0.1.2 (dependency layer keyed on the lock without the project; per-crop embedding). pytest -m gpu tests/vision passes 3/3 on the card, and the canary survived the container recreate.
90 lines
4.3 KiB
Markdown
90 lines
4.3 KiB
Markdown
# restic / esh-ml1
|
|
|
|
ESH GPU LXC (CT 110 on esh-pve). **Outside vzdump on purpose**: everything on it
|
|
is rebuilt from playbooks and stacks. The exception is **augaman's face gallery**
|
|
(biometric data, not reconstructable), and this restic profile exists for it.
|
|
|
|
Writes cross-site to `rest-server-ana` at `10.250.50.70:8000/esh-ml1/`, user
|
|
`esh-ml1`. Wired 2026-09-27 by [`playbooks/esh-ml1-restic.yaml`](../../../playbooks/esh-ml1-restic.yaml).
|
|
|
|
## What's backed up
|
|
|
|
| Path | Purpose |
|
|
|---|---|
|
|
| `/var/lib/restic/stage` | `augaman/gallery.db`, written by `pre-backup.sh` each run |
|
|
| `/opt/docker/compose` | the deployed compose files and `.env`s (a few KB) |
|
|
|
|
Daily **0100 PT**, weekly check Sun **0500 PT**. The CT's clock is UTC, so both
|
|
schedules carry `America/Los_Angeles` explicitly.
|
|
|
|
## pre-backup.sh is fail-closed
|
|
|
|
It runs augaman's own backup CLI (`docker exec augaman python -m
|
|
augaman.gallery.backup ...`: SQLite backup API, `integrity_check`, atomic rename,
|
|
mode 0600), then checks that the staged file is non-empty and under 5 minutes
|
|
old. Any failure, **including the augaman container being stopped**, aborts the
|
|
run: no snapshot, no `last-success`, and the fleet freshness check alerts.
|
|
|
|
Tested by making it fail (2026-09-27): with a stub `docker` that exits 1 on PATH,
|
|
`resticprofile backup` returned 1, the snapshot count stayed 1 → 1, and
|
|
`last-success` did not move.
|
|
|
|
## ⚠ Deviation from the other hosts: `repository-file`, not `restic.env`
|
|
|
|
`resticprofile schedule` copies `env-file` values into the generated systemd
|
|
units, which are world-readable. On the env-file hosts that publishes the
|
|
rest-server password to every local user (see `docs/runbooks/backups.md` →
|
|
Known gaps). Here the URL lives in `/etc/restic/repository` (root 0400) and the
|
|
unit carries only the path. The playbook's verify step fails if a unit ever
|
|
contains `rest:http`.
|
|
|
|
## Secrets (seeded by hand; never in this repo)
|
|
|
|
| host file | vault item |
|
|
|---|---|
|
|
| `/etc/restic/password` (root 0400), encryption passphrase | `esh-ml1/etc/restic/password` |
|
|
| `/etc/restic/repository` (root 0400), `rest:http://esh-ml1:<pw>@10.250.50.70:8000/esh-ml1/` | `esh-ml1/etc/restic/repository` |
|
|
|
|
To re-seed a rebuilt CT from the vault (from nh3-dev):
|
|
|
|
```bash
|
|
printf '%s\n' "$(secret get esh-ml1/etc/restic/password)" \
|
|
| ssh infra-ops@10.0.50.80 'sudo install -d -m 0755 /etc/restic && sudo install -o root -g root -m 0400 /dev/stdin /etc/restic/password'
|
|
printf '%s\n' "$(secret get esh-ml1/etc/restic/repository)" \
|
|
| ssh infra-ops@10.0.50.80 'sudo install -o root -g root -m 0400 /dev/stdin /etc/restic/repository'
|
|
scripts/elway esh-ml1 --playbook playbooks/esh-ml1-restic.yaml
|
|
```
|
|
|
|
The rest-server side is a bcrypt line for `esh-ml1` in
|
|
`/mnt/backup/restic/repo/ana/.htpasswd` on ana-docker. That file is on
|
|
root-squashed NFS, so edit it as `lkraven` (`ssh ana-docker`), not with sudo.
|
|
rest-server picked up the new line without a restart. The NH3 Synology's
|
|
`.htpasswd` does **not** have an `esh-ml1` line yet, so a DR move of this client
|
|
to rest-server-nh3 needs one added first.
|
|
|
|
## Restoring the gallery
|
|
|
|
```bash
|
|
sudo -n bash -c 'R="restic --repository-file /etc/restic/repository --password-file /etc/restic/password"
|
|
t=$(mktemp -d); $R restore latest --target $t --include /var/lib/restic/stage/augaman/gallery.db
|
|
ls -l $t/var/lib/restic/stage/augaman/gallery.db'
|
|
```
|
|
|
|
The live gallery is the `augaman_gallery` named volume (`/data/gallery.db` in
|
|
the container). Putting a restored copy back is augaman's procedure; ask
|
|
augaman-dev before overwriting a live gallery.
|
|
|
|
**Verified 2026-09-27, two ways.**
|
|
- **Mechanics**, against snapshot `2685d3f6` (empty gallery): the restored file
|
|
was byte-identical to the staged one (sha256), and `integrity_check` was ok.
|
|
- **Identities**, against snapshot `fd3061a1`, taken while augaman-dev's
|
|
public-domain canary was enrolled (1 identity, 3 samples). The restored copy
|
|
and the live gallery gave the same sha256 digest over every identity row
|
|
(id, name_key, created_at) and every sample (id, embedder, embedding), and
|
|
`integrity_check` was ok.
|
|
|
|
Together these satisfy the operator ruling's gate (a scheduled off-box backup
|
|
plus one verified restore) before real enrollments. To re-run the check, digest
|
|
both copies the same way. The empty snapshot `2685d3f6` is a ready-made
|
|
negative control: it must NOT match a non-empty live gallery.
|