# restic / esh-ml1

ESH GPU LXC (CT 110 on esh-pve). **Outside vzdump on purpose**: everything on it
is rebuilt from playbooks and stacks. The exception is **augaman's face gallery**
(biometric data, not reconstructable), and this restic profile exists for it.

Writes cross-site to `rest-server-ana` at `10.250.50.70:8000/esh-ml1/`, user
`esh-ml1`. Wired 2026-09-27 by [`playbooks/esh-ml1-restic.yaml`](../../../playbooks/esh-ml1-restic.yaml).

## What's backed up

| Path | Purpose |
|---|---|
| `/var/lib/restic/stage` | `augaman/gallery.db`, written by `pre-backup.sh` each run |
| `/opt/docker/compose` | the deployed compose files and `.env`s (a few KB) |

Daily **0100 PT**, weekly check Sun **0500 PT**. The CT's clock is UTC, so both
schedules carry `America/Los_Angeles` explicitly.

## pre-backup.sh is fail-closed

It runs augaman's own backup CLI (`docker exec augaman python -m
augaman.gallery.backup ...`: SQLite backup API, `integrity_check`, atomic rename,
mode 0600), then checks that the staged file is non-empty and under 5 minutes
old. Any failure, **including the augaman container being stopped**, aborts the
run: no snapshot, no `last-success`, and the fleet freshness check alerts.

Tested by making it fail (2026-09-27): with a stub `docker` that exits 1 on PATH,
`resticprofile backup` returned 1, the snapshot count stayed 1 → 1, and
`last-success` did not move.

## ⚠ Deviation from the other hosts: `repository-file`, not `restic.env`

`resticprofile schedule` copies `env-file` values into the generated systemd
units, which are world-readable. On the env-file hosts that publishes the
rest-server password to every local user (see `docs/runbooks/backups.md` →
Known gaps). Here the URL lives in `/etc/restic/repository` (root 0400) and the
unit carries only the path. The playbook's verify step fails if a unit ever
contains `rest:http`.

## Secrets (seeded by hand; never in this repo)

| host file | vault item |
|---|---|
| `/etc/restic/password` (root 0400), encryption passphrase | `esh-ml1/etc/restic/password` |
| `/etc/restic/repository` (root 0400), `rest:http://esh-ml1:<pw>@10.250.50.70:8000/esh-ml1/` | `esh-ml1/etc/restic/repository` |

To re-seed a rebuilt CT from the vault (from nh3-dev):

```bash
printf '%s\n' "$(secret get esh-ml1/etc/restic/password)" \
  | ssh infra-ops@10.0.50.80 'sudo install -d -m 0755 /etc/restic && sudo install -o root -g root -m 0400 /dev/stdin /etc/restic/password'
printf '%s\n' "$(secret get esh-ml1/etc/restic/repository)" \
  | ssh infra-ops@10.0.50.80 'sudo install -o root -g root -m 0400 /dev/stdin /etc/restic/repository'
scripts/elway esh-ml1 --playbook playbooks/esh-ml1-restic.yaml
```

The rest-server side is a bcrypt line for `esh-ml1` in
`/mnt/backup/restic/repo/ana/.htpasswd` on ana-docker. That file is on
root-squashed NFS, so edit it as `lkraven` (`ssh ana-docker`), not with sudo.
rest-server picked up the new line without a restart. The NH3 Synology's
`.htpasswd` does **not** have an `esh-ml1` line yet, so a DR move of this client
to rest-server-nh3 needs one added first.

## Restoring the gallery

```bash
sudo -n bash -c 'R="restic --repository-file /etc/restic/repository --password-file /etc/restic/password"
  t=$(mktemp -d); $R restore latest --target $t --include /var/lib/restic/stage/augaman/gallery.db
  ls -l $t/var/lib/restic/stage/augaman/gallery.db'
```

The live gallery is the `augaman_gallery` named volume (`/data/gallery.db` in
the container). Putting a restored copy back is augaman's procedure; ask
augaman-dev before overwriting a live gallery.

**Verified 2026-09-27, two ways.**
- **Mechanics**, against snapshot `2685d3f6` (empty gallery): the restored file
  was byte-identical to the staged one (sha256), and `integrity_check` was ok.
- **Identities**, against snapshot `fd3061a1`, taken while augaman-dev's
  public-domain canary was enrolled (1 identity, 3 samples). The restored copy
  and the live gallery gave the same sha256 digest over every identity row
  (id, name_key, created_at) and every sample (id, embedder, embedding), and
  `integrity_check` was ok.

Together these satisfy the operator ruling's gate (a scheduled off-box backup
plus one verified restore) before real enrollments. To re-run the check, digest
both copies the same way. The empty snapshot `2685d3f6` is a ready-made
negative control: it must NOT match a non-empty live gallery.