feat(esh-ml1): restic backup of augaman's gallery; augaman v0.1.2

esh-ml1 is outside vzdump, so augaman's face gallery reaches backup only
through restic. New playbooks/esh-ml1-restic.yaml installs restic 0.14.0 (the
same Debian package as the other ESH hosts) and resticprofile 0.33.1 (pinned,
sha256-checked). It uploads configs/restic/esh-ml1/ and schedules a daily
0100 PT backup plus a Sunday 0500 PT check to rest-server-ana. The CT runs UTC,
so both schedules name the zone explicitly.

pre-backup.sh is fail-closed: it runs augaman's own backup CLI, and any failure,
including a stopped container, aborts the run. Tested with a stub docker that
exits 1: the run returned 1, and neither the snapshot count nor last-success
moved. The restore was verified at identity level against augaman-dev's
public-domain canary (snapshot fd3061a1: the restored copy's digest over
identities and samples matches the live gallery). That meets the operator gate
for real enrollments.

The repository URL is read through repository-file rather than restic.env.
resticprofile schedule copies env-file values into world-readable systemd
units, which publishes the rest-server password on the env-file hosts
(observed on esh-docker-vm). This is recorded in the backups runbook under
Known gaps, and the playbook verifies no generated unit contains the URL.

esh-ml1 is added to the freshness check's expected ana-side repos and to the
runbook tables.

augaman moves to v0.1.2 (dependency layer keyed on the lock without the
project; per-crop embedding). pytest -m gpu tests/vision passes 3/3 on the
card, and the canary survived the container recreate.
This commit is contained in:
vh
2026-09-27 00:06:33 -07:00
parent c26f7c94e4
commit d8f59a15d9
10 changed files with 347 additions and 18 deletions
+22 -2
View File
@@ -29,6 +29,7 @@ layer has failed.
| nh3-docker | VM (nh3-pve) | ✅ `nh3-pve` | ✅ → rest-server-**nh3** | no |
| esh-docker-vm | VM (esh-pve) | ✅ `esh-pve` | ✅ → rest-server-**ana** | no |
| esh-vm-db | VM (esh-pve-nas) | ❌ **none** (esh-pve-nas not a PBS source) | ✅ → rest-server-**ana** | ⚠️ **restic-only (a DB!)** |
| **esh-ml1** | CT 110 (esh-pve) | ❌ none (**outside vzdump on purpose**: rebuilt from playbooks) | ✅ → rest-server-**ana** (augaman gallery + compose dir only) | ⚠️ **restic is the ONLY net for augaman's biometric gallery** |
| vm-esh-nas | VM (esh-pve-nas) | ❌ **none** (esh-pve-nas not a PBS source) | ✅ → rest-server-**ana** | ⚠️ **restic-only** |
| other pfi-pve / nh3-pve VMs/CTs | VM/CT | ✅ respective ns | (PBS only) | no |
| SureFire `sfsrv-pve` | tenant VMs | ✅ `sfsrv-pve` ns | (PBS only) | no |
@@ -64,7 +65,7 @@ auth, append-only, private repos). The split is by site:
| rest-server | Endpoint | Backing store | Clients |
|---|---|---|---|
| **rest-server-ana** | `http://10.250.50.70:8000` (container `rest-server` on ana-docker) | `ana-nas:/mnt/backup/restic/repo/ana` (NFS bind → `/data`) | ana-docker, **ana-ml2**, esh-docker-vm, esh-vm-db, vm-esh-nas |
| **rest-server-ana** | `http://10.250.50.70:8000` (container `rest-server` on ana-docker) | `ana-nas:/mnt/backup/restic/repo/ana` (NFS bind → `/data`) | ana-docker, **ana-ml2**, esh-docker-vm, esh-ml1, esh-vm-db, vm-esh-nas |
| **rest-server-nh3** | `http://10.100.50.50:8000` (on nh3-nas) | `nh3-nas:/volume1/Backup/restic/<client>` | **irv-ml1**, nh3-docker |
- Per-client repos live as subdirs of the rest-server data dir
@@ -115,7 +116,7 @@ Run these any time you need to answer "are we backed up?"
```bash
# --- restic ANA side: newest snapshot per client (want: today/yesterday) ---
ssh ana-nas 'for c in ana-docker ana-ml2 esh-docker-vm esh-vm-db vm-esh-nas; do
ssh ana-nas 'for c in ana-docker ana-ml2 esh-docker-vm esh-ml1 esh-vm-db vm-esh-nas; do
echo -n "$c: "; ls -t /mnt/backup/restic/repo/ana/$c/snapshots/ 2>/dev/null | head -1 \
| xargs -I{} stat -c "%y" /mnt/backup/restic/repo/ana/$c/snapshots/{} 2>/dev/null || echo MISSING
done'
@@ -177,6 +178,25 @@ the stop→umount→rm-ghost→remount→start variant.
## Known gaps / TODO
### ⚠ `resticprofile schedule` publishes the repo credential (found 2026-09-27)
On a host whose profile loads `RESTIC_REPOSITORY` from `env-file:
/etc/restic/restic.env`, `resticprofile schedule` copies that value, **including
the embedded rest-server basic-auth password**, into
`/etc/systemd/system/resticprofile-*@profile-default.service` as an
`Environment=` line. Those units are world-readable, so `systemctl cat` shows the
credential to any local user; the 0600 on `restic.env` protects nothing.
Observed on esh-docker-vm (read without sudo). Blast radius: the credential lets
a caller read the encrypted blobs and append to that one repo (rest-server is
`--append-only`, and the encryption passphrase is a separate file). It does not
decrypt anything.
**esh-ml1 does not have this problem:** its profile uses `repository-file:
/etc/restic/repository` (root 0400), so the unit carries only the path, and
`playbooks/esh-ml1-restic.yaml` verifies that no unit contains `rest:http`.
Moving the other hosts to `repository-file` is not done; it belongs with the
(operator-held) rest-server password rotation below.
### ✅ restic content assertion (2026-09-22, CLOSED)
⚠ **I reported this gap wrongly first.** I ran `grep -ic restic` against