feat(esh-ml1): restic backup of augaman's gallery; augaman v0.1.2
esh-ml1 is outside vzdump, so augaman's face gallery reaches backup only through restic. New playbooks/esh-ml1-restic.yaml installs restic 0.14.0 (the same Debian package as the other ESH hosts) and resticprofile 0.33.1 (pinned, sha256-checked). It uploads configs/restic/esh-ml1/ and schedules a daily 0100 PT backup plus a Sunday 0500 PT check to rest-server-ana. The CT runs UTC, so both schedules name the zone explicitly. pre-backup.sh is fail-closed: it runs augaman's own backup CLI, and any failure, including a stopped container, aborts the run. Tested with a stub docker that exits 1: the run returned 1, and neither the snapshot count nor last-success moved. The restore was verified at identity level against augaman-dev's public-domain canary (snapshot fd3061a1: the restored copy's digest over identities and samples matches the live gallery). That meets the operator gate for real enrollments. The repository URL is read through repository-file rather than restic.env. resticprofile schedule copies env-file values into world-readable systemd units, which publishes the rest-server password on the env-file hosts (observed on esh-docker-vm). This is recorded in the backups runbook under Known gaps, and the playbook verifies no generated unit contains the URL. esh-ml1 is added to the freshness check's expected ana-side repos and to the runbook tables. augaman moves to v0.1.2 (dependency layer keyed on the lock without the project; per-crop embedding). pytest -m gpu tests/vision passes 3/3 on the card, and the canary survived the container recreate.
This commit is contained in:
@@ -29,6 +29,7 @@ layer has failed.
|
||||
| nh3-docker | VM (nh3-pve) | ✅ `nh3-pve` | ✅ → rest-server-**nh3** | no |
|
||||
| esh-docker-vm | VM (esh-pve) | ✅ `esh-pve` | ✅ → rest-server-**ana** | no |
|
||||
| esh-vm-db | VM (esh-pve-nas) | ❌ **none** (esh-pve-nas not a PBS source) | ✅ → rest-server-**ana** | ⚠️ **restic-only (a DB!)** |
|
||||
| **esh-ml1** | CT 110 (esh-pve) | ❌ none (**outside vzdump on purpose**: rebuilt from playbooks) | ✅ → rest-server-**ana** (augaman gallery + compose dir only) | ⚠️ **restic is the ONLY net for augaman's biometric gallery** |
|
||||
| vm-esh-nas | VM (esh-pve-nas) | ❌ **none** (esh-pve-nas not a PBS source) | ✅ → rest-server-**ana** | ⚠️ **restic-only** |
|
||||
| other pfi-pve / nh3-pve VMs/CTs | VM/CT | ✅ respective ns | (PBS only) | no |
|
||||
| SureFire `sfsrv-pve` | tenant VMs | ✅ `sfsrv-pve` ns | (PBS only) | no |
|
||||
@@ -64,7 +65,7 @@ auth, append-only, private repos). The split is by site:
|
||||
|
||||
| rest-server | Endpoint | Backing store | Clients |
|
||||
|---|---|---|---|
|
||||
| **rest-server-ana** | `http://10.250.50.70:8000` (container `rest-server` on ana-docker) | `ana-nas:/mnt/backup/restic/repo/ana` (NFS bind → `/data`) | ana-docker, **ana-ml2**, esh-docker-vm, esh-vm-db, vm-esh-nas |
|
||||
| **rest-server-ana** | `http://10.250.50.70:8000` (container `rest-server` on ana-docker) | `ana-nas:/mnt/backup/restic/repo/ana` (NFS bind → `/data`) | ana-docker, **ana-ml2**, esh-docker-vm, esh-ml1, esh-vm-db, vm-esh-nas |
|
||||
| **rest-server-nh3** | `http://10.100.50.50:8000` (on nh3-nas) | `nh3-nas:/volume1/Backup/restic/<client>` | **irv-ml1**, nh3-docker |
|
||||
|
||||
- Per-client repos live as subdirs of the rest-server data dir
|
||||
@@ -115,7 +116,7 @@ Run these any time you need to answer "are we backed up?"
|
||||
|
||||
```bash
|
||||
# --- restic ANA side: newest snapshot per client (want: today/yesterday) ---
|
||||
ssh ana-nas 'for c in ana-docker ana-ml2 esh-docker-vm esh-vm-db vm-esh-nas; do
|
||||
ssh ana-nas 'for c in ana-docker ana-ml2 esh-docker-vm esh-ml1 esh-vm-db vm-esh-nas; do
|
||||
echo -n "$c: "; ls -t /mnt/backup/restic/repo/ana/$c/snapshots/ 2>/dev/null | head -1 \
|
||||
| xargs -I{} stat -c "%y" /mnt/backup/restic/repo/ana/$c/snapshots/{} 2>/dev/null || echo MISSING
|
||||
done'
|
||||
@@ -177,6 +178,25 @@ the stop→umount→rm-ghost→remount→start variant.
|
||||
|
||||
## Known gaps / TODO
|
||||
|
||||
### ⚠ `resticprofile schedule` publishes the repo credential (found 2026-09-27)
|
||||
|
||||
On a host whose profile loads `RESTIC_REPOSITORY` from `env-file:
|
||||
/etc/restic/restic.env`, `resticprofile schedule` copies that value, **including
|
||||
the embedded rest-server basic-auth password**, into
|
||||
`/etc/systemd/system/resticprofile-*@profile-default.service` as an
|
||||
`Environment=` line. Those units are world-readable, so `systemctl cat` shows the
|
||||
credential to any local user; the 0600 on `restic.env` protects nothing.
|
||||
Observed on esh-docker-vm (read without sudo). Blast radius: the credential lets
|
||||
a caller read the encrypted blobs and append to that one repo (rest-server is
|
||||
`--append-only`, and the encryption passphrase is a separate file). It does not
|
||||
decrypt anything.
|
||||
|
||||
**esh-ml1 does not have this problem:** its profile uses `repository-file:
|
||||
/etc/restic/repository` (root 0400), so the unit carries only the path, and
|
||||
`playbooks/esh-ml1-restic.yaml` verifies that no unit contains `rest:http`.
|
||||
Moving the other hosts to `repository-file` is not done; it belongs with the
|
||||
(operator-held) rest-server password rotation below.
|
||||
|
||||
### ✅ restic content assertion (2026-09-22, CLOSED)
|
||||
|
||||
⚠ **I reported this gap wrongly first.** I ran `grep -ic restic` against
|
||||
|
||||
Reference in New Issue
Block a user