From d8f59a15d99c074644f9cd7284dab45430658cb6 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 27 Sep 2026 00:06:33 -0700 Subject: [PATCH] feat(esh-ml1): restic backup of augaman's gallery; augaman v0.1.2 esh-ml1 is outside vzdump, so augaman's face gallery reaches backup only through restic. New playbooks/esh-ml1-restic.yaml installs restic 0.14.0 (the same Debian package as the other ESH hosts) and resticprofile 0.33.1 (pinned, sha256-checked). It uploads configs/restic/esh-ml1/ and schedules a daily 0100 PT backup plus a Sunday 0500 PT check to rest-server-ana. The CT runs UTC, so both schedules name the zone explicitly. pre-backup.sh is fail-closed: it runs augaman's own backup CLI, and any failure, including a stopped container, aborts the run. Tested with a stub docker that exits 1: the run returned 1, and neither the snapshot count nor last-success moved. The restore was verified at identity level against augaman-dev's public-domain canary (snapshot fd3061a1: the restored copy's digest over identities and samples matches the live gallery). That meets the operator gate for real enrollments. The repository URL is read through repository-file rather than restic.env. resticprofile schedule copies env-file values into world-readable systemd units, which publishes the rest-server password on the env-file hosts (observed on esh-docker-vm). This is recorded in the backups runbook under Known gaps, and the playbook verifies no generated unit contains the URL. esh-ml1 is added to the freshness check's expected ana-side repos and to the runbook tables. augaman moves to v0.1.2 (dependency layer keyed on the lock without the project; per-crop embedding). pytest -m gpu tests/vision passes 3/3 on the card, and the canary survived the container recreate. --- configs/restic/esh-ml1/README.md | 89 +++++++++++++++++++++++ configs/restic/esh-ml1/pre-backup.sh | 35 +++++++++ configs/restic/esh-ml1/profiles.yaml | 71 ++++++++++++++++++ docs/runbooks/backups.md | 24 ++++++- persistent-memory.md | 14 ++-- playbooks/esh-ml1-restic.yaml | 103 +++++++++++++++++++++++++++ scripts/check-backup-freshness.sh | 2 +- servers/esh-ml1/README.md | 2 +- stacks/augaman/.env.example | 2 +- stacks/augaman/README.md | 23 ++++-- 10 files changed, 347 insertions(+), 18 deletions(-) create mode 100644 configs/restic/esh-ml1/README.md create mode 100755 configs/restic/esh-ml1/pre-backup.sh create mode 100644 configs/restic/esh-ml1/profiles.yaml create mode 100644 playbooks/esh-ml1-restic.yaml diff --git a/configs/restic/esh-ml1/README.md b/configs/restic/esh-ml1/README.md new file mode 100644 index 0000000..535355e --- /dev/null +++ b/configs/restic/esh-ml1/README.md @@ -0,0 +1,89 @@ +# restic / esh-ml1 + +ESH GPU LXC (CT 110 on esh-pve). **Outside vzdump on purpose**: everything on it +is rebuilt from playbooks and stacks. The exception is **augaman's face gallery** +(biometric data, not reconstructable), and this restic profile exists for it. + +Writes cross-site to `rest-server-ana` at `10.250.50.70:8000/esh-ml1/`, user +`esh-ml1`. Wired 2026-09-27 by [`playbooks/esh-ml1-restic.yaml`](../../../playbooks/esh-ml1-restic.yaml). + +## What's backed up + +| Path | Purpose | +|---|---| +| `/var/lib/restic/stage` | `augaman/gallery.db`, written by `pre-backup.sh` each run | +| `/opt/docker/compose` | the deployed compose files and `.env`s (a few KB) | + +Daily **0100 PT**, weekly check Sun **0500 PT**. The CT's clock is UTC, so both +schedules carry `America/Los_Angeles` explicitly. + +## pre-backup.sh is fail-closed + +It runs augaman's own backup CLI (`docker exec augaman python -m +augaman.gallery.backup ...`: SQLite backup API, `integrity_check`, atomic rename, +mode 0600), then checks that the staged file is non-empty and under 5 minutes +old. Any failure, **including the augaman container being stopped**, aborts the +run: no snapshot, no `last-success`, and the fleet freshness check alerts. + +Tested by making it fail (2026-09-27): with a stub `docker` that exits 1 on PATH, +`resticprofile backup` returned 1, the snapshot count stayed 1 → 1, and +`last-success` did not move. + +## ⚠ Deviation from the other hosts: `repository-file`, not `restic.env` + +`resticprofile schedule` copies `env-file` values into the generated systemd +units, which are world-readable. On the env-file hosts that publishes the +rest-server password to every local user (see `docs/runbooks/backups.md` → +Known gaps). Here the URL lives in `/etc/restic/repository` (root 0400) and the +unit carries only the path. The playbook's verify step fails if a unit ever +contains `rest:http`. + +## Secrets (seeded by hand; never in this repo) + +| host file | vault item | +|---|---| +| `/etc/restic/password` (root 0400), encryption passphrase | `esh-ml1/etc/restic/password` | +| `/etc/restic/repository` (root 0400), `rest:http://esh-ml1:@10.250.50.70:8000/esh-ml1/` | `esh-ml1/etc/restic/repository` | + +To re-seed a rebuilt CT from the vault (from nh3-dev): + +```bash +printf '%s\n' "$(secret get esh-ml1/etc/restic/password)" \ + | ssh infra-ops@10.0.50.80 'sudo install -d -m 0755 /etc/restic && sudo install -o root -g root -m 0400 /dev/stdin /etc/restic/password' +printf '%s\n' "$(secret get esh-ml1/etc/restic/repository)" \ + | ssh infra-ops@10.0.50.80 'sudo install -o root -g root -m 0400 /dev/stdin /etc/restic/repository' +scripts/elway esh-ml1 --playbook playbooks/esh-ml1-restic.yaml +``` + +The rest-server side is a bcrypt line for `esh-ml1` in +`/mnt/backup/restic/repo/ana/.htpasswd` on ana-docker. That file is on +root-squashed NFS, so edit it as `lkraven` (`ssh ana-docker`), not with sudo. +rest-server picked up the new line without a restart. The NH3 Synology's +`.htpasswd` does **not** have an `esh-ml1` line yet, so a DR move of this client +to rest-server-nh3 needs one added first. + +## Restoring the gallery + +```bash +sudo -n bash -c 'R="restic --repository-file /etc/restic/repository --password-file /etc/restic/password" + t=$(mktemp -d); $R restore latest --target $t --include /var/lib/restic/stage/augaman/gallery.db + ls -l $t/var/lib/restic/stage/augaman/gallery.db' +``` + +The live gallery is the `augaman_gallery` named volume (`/data/gallery.db` in +the container). Putting a restored copy back is augaman's procedure; ask +augaman-dev before overwriting a live gallery. + +**Verified 2026-09-27, two ways.** +- **Mechanics**, against snapshot `2685d3f6` (empty gallery): the restored file + was byte-identical to the staged one (sha256), and `integrity_check` was ok. +- **Identities**, against snapshot `fd3061a1`, taken while augaman-dev's + public-domain canary was enrolled (1 identity, 3 samples). The restored copy + and the live gallery gave the same sha256 digest over every identity row + (id, name_key, created_at) and every sample (id, embedder, embedding), and + `integrity_check` was ok. + +Together these satisfy the operator ruling's gate (a scheduled off-box backup +plus one verified restore) before real enrollments. To re-run the check, digest +both copies the same way. The empty snapshot `2685d3f6` is a ready-made +negative control: it must NOT match a non-empty live gallery. diff --git a/configs/restic/esh-ml1/pre-backup.sh b/configs/restic/esh-ml1/pre-backup.sh new file mode 100755 index 0000000..ff4bca6 --- /dev/null +++ b/configs/restic/esh-ml1/pre-backup.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# pre-backup hook for esh-ml1 — deployed to /etc/restic/pre-backup.sh (root, 0755). +# +# Takes an online-consistent copy of augaman's face gallery with augaman's own +# backup CLI (SQLite backup API, integrity_check, atomic rename, mode 0600) into +# /var/lib/restic/stage/augaman/gallery.db, which restic then snapshots. +# +# FAIL-CLOSED. Any failure exits non-zero, which makes resticprofile abort the +# run: no snapshot, no last-success stamp, and the fleet freshness check alerts. +# That includes the augaman container being stopped. A gallery that is not being +# backed up must be loud, not a quietly stale file in every snapshot. + +set -euo pipefail + +STAGE_FILE=/var/lib/restic/stage/augaman/gallery.db +MAX_AGE_S=300 + +if ! out=$(docker exec augaman python -m augaman.gallery.backup \ + --db /data/gallery.db --dest /backup/gallery.db 2>&1); then + echo "pre-backup: augaman gallery backup FAILED: ${out}" >&2 + exit 1 +fi +echo "pre-backup: augaman gallery backup: ${out}" + +# The CLI said it succeeded; confirm the file restic is about to read is the +# one it just wrote (non-empty, and written within the last few minutes). +if [ ! -s "$STAGE_FILE" ]; then + echo "pre-backup: ${STAGE_FILE} is missing or empty after a successful CLI run" >&2 + exit 1 +fi +age=$(( $(date +%s) - $(stat -c %Y "$STAGE_FILE") )) +if [ "$age" -gt "$MAX_AGE_S" ]; then + echo "pre-backup: ${STAGE_FILE} is ${age}s old; the CLI did not refresh it" >&2 + exit 1 +fi diff --git a/configs/restic/esh-ml1/profiles.yaml b/configs/restic/esh-ml1/profiles.yaml new file mode 100644 index 0000000..30d3fcc --- /dev/null +++ b/configs/restic/esh-ml1/profiles.yaml @@ -0,0 +1,71 @@ +# resticprofile config for esh-ml1. +# +# ESH GPU LXC (CT 110 on esh-pve). The host is OUTSIDE vzdump on purpose: +# everything on it is rebuilt from playbooks + stacks, and models re-download. +# The one exception is augaman's face gallery (biometric data, not +# reconstructable), so this profile exists for that and little else. +# +# Writes cross-site to rest-server-ana (10.250.50.70:8000/esh-ml1/) because +# the ESH site has no local rest-server (same as esh-docker-vm / esh-vm-db). +# +# ⚠ DELIBERATE DEVIATION from the other hosts: the repository URL (which +# embeds the rest-server basic-auth password) is read from +# /etc/restic/repository via `repository-file`, NOT from an env-file. +# `resticprofile schedule` copies env-file values into the generated systemd +# unit, and units under /etc/systemd/system are world-readable (0644), so +# the env-file pattern publishes the credential to every local user. With +# repository-file the unit carries only the path. See README.md. +# +# ⚠ The CT runs in UTC; the schedules carry an explicit zone so they fire at +# the same wall-clock time as the rest of the fleet. + +version: "1" + +global: + priority: low + ionice: true + ionice-class: 2 + ionice-level: 7 + min-memory: 100 + +default: + repository-file: /etc/restic/repository + password-file: /etc/restic/password + initialize: false + lock: /var/lock/restic-esh-ml1.lock + + backup: + verbose: 1 + run-before: + # Fail-closed: if augaman's backup CLI fails (or the container is down), + # the whole run fails, last-success is not written, and the freshness + # check alerts. A stale gallery copy must never be snapshotted as current. + - /etc/restic/pre-backup.sh + run-after: + - date +%s > /var/lib/restic/last-success + source: + - /var/lib/restic/stage # augaman/gallery.db, written by pre-backup.sh + - /opt/docker/compose # the deployed compose files + .env (small) + tag: + - host:esh-ml1 + - site:esh + - fleet:home-lab + schedule: "*-*-* 01:00:00 America/Los_Angeles" + schedule-permission: system + schedule-log: /var/log/restic-backup.log + + forget: + keep-daily: 7 + keep-weekly: 4 + keep-monthly: 12 + keep-yearly: 3 + tag: + - host:esh-ml1 + # No schedule: rest-server-ana is --append-only, so forget always fails + # from the client. It runs by hand during the prune ceremony. + + check: + read-data-subset: 10% + schedule: "Sun *-*-* 05:00:00 America/Los_Angeles" + schedule-permission: system + schedule-log: /var/log/restic-check.log diff --git a/docs/runbooks/backups.md b/docs/runbooks/backups.md index 4164f8c..ea6e3e6 100644 --- a/docs/runbooks/backups.md +++ b/docs/runbooks/backups.md @@ -29,6 +29,7 @@ layer has failed. | nh3-docker | VM (nh3-pve) | ✅ `nh3-pve` | ✅ → rest-server-**nh3** | no | | esh-docker-vm | VM (esh-pve) | ✅ `esh-pve` | ✅ → rest-server-**ana** | no | | esh-vm-db | VM (esh-pve-nas) | ❌ **none** (esh-pve-nas not a PBS source) | ✅ → rest-server-**ana** | ⚠️ **restic-only (a DB!)** | +| **esh-ml1** | CT 110 (esh-pve) | ❌ none (**outside vzdump on purpose**: rebuilt from playbooks) | ✅ → rest-server-**ana** (augaman gallery + compose dir only) | ⚠️ **restic is the ONLY net for augaman's biometric gallery** | | vm-esh-nas | VM (esh-pve-nas) | ❌ **none** (esh-pve-nas not a PBS source) | ✅ → rest-server-**ana** | ⚠️ **restic-only** | | other pfi-pve / nh3-pve VMs/CTs | VM/CT | ✅ respective ns | (PBS only) | no | | SureFire `sfsrv-pve` | tenant VMs | ✅ `sfsrv-pve` ns | (PBS only) | no | @@ -64,7 +65,7 @@ auth, append-only, private repos). The split is by site: | rest-server | Endpoint | Backing store | Clients | |---|---|---|---| -| **rest-server-ana** | `http://10.250.50.70:8000` (container `rest-server` on ana-docker) | `ana-nas:/mnt/backup/restic/repo/ana` (NFS bind → `/data`) | ana-docker, **ana-ml2**, esh-docker-vm, esh-vm-db, vm-esh-nas | +| **rest-server-ana** | `http://10.250.50.70:8000` (container `rest-server` on ana-docker) | `ana-nas:/mnt/backup/restic/repo/ana` (NFS bind → `/data`) | ana-docker, **ana-ml2**, esh-docker-vm, esh-ml1, esh-vm-db, vm-esh-nas | | **rest-server-nh3** | `http://10.100.50.50:8000` (on nh3-nas) | `nh3-nas:/volume1/Backup/restic/` | **irv-ml1**, nh3-docker | - Per-client repos live as subdirs of the rest-server data dir @@ -115,7 +116,7 @@ Run these any time you need to answer "are we backed up?" ```bash # --- restic ANA side: newest snapshot per client (want: today/yesterday) --- -ssh ana-nas 'for c in ana-docker ana-ml2 esh-docker-vm esh-vm-db vm-esh-nas; do +ssh ana-nas 'for c in ana-docker ana-ml2 esh-docker-vm esh-ml1 esh-vm-db vm-esh-nas; do echo -n "$c: "; ls -t /mnt/backup/restic/repo/ana/$c/snapshots/ 2>/dev/null | head -1 \ | xargs -I{} stat -c "%y" /mnt/backup/restic/repo/ana/$c/snapshots/{} 2>/dev/null || echo MISSING done' @@ -177,6 +178,25 @@ the stop→umount→rm-ghost→remount→start variant. ## Known gaps / TODO +### ⚠ `resticprofile schedule` publishes the repo credential (found 2026-09-27) + +On a host whose profile loads `RESTIC_REPOSITORY` from `env-file: +/etc/restic/restic.env`, `resticprofile schedule` copies that value, **including +the embedded rest-server basic-auth password**, into +`/etc/systemd/system/resticprofile-*@profile-default.service` as an +`Environment=` line. Those units are world-readable, so `systemctl cat` shows the +credential to any local user; the 0600 on `restic.env` protects nothing. +Observed on esh-docker-vm (read without sudo). Blast radius: the credential lets +a caller read the encrypted blobs and append to that one repo (rest-server is +`--append-only`, and the encryption passphrase is a separate file). It does not +decrypt anything. + +**esh-ml1 does not have this problem:** its profile uses `repository-file: +/etc/restic/repository` (root 0400), so the unit carries only the path, and +`playbooks/esh-ml1-restic.yaml` verifies that no unit contains `rest:http`. +Moving the other hosts to `repository-file` is not done; it belongs with the +(operator-held) rest-server password rotation below. + ### ✅ restic content assertion (2026-09-22, CLOSED) ⚠ **I reported this gap wrongly first.** I ran `grep -ic restic` against diff --git a/persistent-memory.md b/persistent-memory.md index faac2ef..a106f55 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -144,13 +144,13 @@ _As of 2026-09-26 ~1620 PT._ ### augaman: face recognition for Cicada -- **v0.1.1 DEPLOYED on esh-ml1:8040 2026-09-26 2347 PT** (`stacks/augaman`), - healthy on CUDA, in `nvidia-smi`. Built on-box from a `git archive` of the tag. -- **infra-ops owes the gallery backup NEXT:** esh-ml1 has no restic. - `BACKUP_DIR=/var/lib/restic/stage/augaman` exists; needs restic + timer running - the backup CLI, then one verified restore. **Fixtures only until then** (Prime). -- `pytest -m gpu` FAILS 3/3 on the batch-vs-single tolerance (≥6e-5 vs 1e-5); - placement passes. Reported to augaman-dev: the contract is theirs to rule on. +- **v0.1.2 LIVE on esh-ml1:8040** (2026-09-27; v0.1.1 first deployed 2026-09-26 2347 PT), + `stacks/augaman`, healthy on CUDA, in `nvidia-smi`. Built on-box from a `git archive` + of the tag. `pytest -m gpu tests/vision` 3/3 PASS on v0.1.2. +- **Gallery backup WIRED + RESTORE-VERIFIED 2026-09-27** (Prime OK'd off-site): restic + daily 0100 PT → rest-server-ana `esh-ml1/`, fail-closed hook, secrets vaulted under + `esh-ml1/etc/restic/`. Identity-level restore matched the live gallery (canary, + snapshot `fd3061a1`). **The backup gate for real enrollments is MET.** ### esh-ml1 diff --git a/playbooks/esh-ml1-restic.yaml b/playbooks/esh-ml1-restic.yaml new file mode 100644 index 0000000..dbf97d0 --- /dev/null +++ b/playbooks/esh-ml1-restic.yaml @@ -0,0 +1,103 @@ +# esh-ml1 restic: backs up augaman's face gallery (and the deployed compose dir) +# to rest-server-ana. Config + rationale: configs/restic/esh-ml1/. +# +# Run: scripts/elway esh-ml1 --playbook playbooks/esh-ml1-restic.yaml +# +# PRE-REQUISITE, NOT DONE HERE (secrets never live in this repo): seed the two +# root-only secret files from the vault, and the rest-server-ana htpasswd entry +# for user `esh-ml1`. The exact commands are in configs/restic/esh-ml1/README.md. +# The "Secrets are seeded" step below halts the run if they are missing. +# +# Idempotent: a second run should report ok/skipped everywhere except the +# re-uploads and the schedule refresh. + +vars: + resticprofile_version: 0.33.1 + resticprofile_sha256: 5fecd20de21811a750a4d61e841b2258fdf018bbfffad136c96078df27b452df + +steps: + - name: Install restic (Debian 12 package, 0.14.0, same as esh-docker-vm and esh-vm-db) + sudo: true + shell: DEBIAN_FRONTEND=noninteractive apt-get install -y -q restic + when: "! command -v restic >/dev/null" + + - name: Install resticprofile {{ resticprofile_version }} (pinned, sha256-checked) + sudo: true + shell: | + set -eu + tmp=$(mktemp -d) + trap 'rm -rf "$tmp"' EXIT + tgz="resticprofile_{{ resticprofile_version }}_linux_amd64.tar.gz" + curl -fsSL -o "$tmp/$tgz" "https://github.com/creativeprojects/resticprofile/releases/download/v{{ resticprofile_version }}/$tgz" + echo "{{ resticprofile_sha256 }} $tmp/$tgz" | sha256sum -c - + tar -xzf "$tmp/$tgz" -C "$tmp" resticprofile + install -o root -g root -m 0755 "$tmp/resticprofile" /usr/local/bin/resticprofile + when: "! /usr/local/bin/resticprofile version 2>/dev/null | grep -q 'version {{ resticprofile_version }} '" + + - name: Directories (stage is root-only; its augaman subdir belongs to the container user 10001) + sudo: true + shell: | + set -eu + install -d -o root -g root -m 0755 /etc/restic + install -d -o root -g root -m 0700 /var/lib/restic/stage + install -d -o 10001 -g 10001 -m 0700 /var/lib/restic/stage/augaman + when: "! sudo -n sh -c 'test \"$(stat -c %U:%a /var/lib/restic/stage)\" = root:700 && test \"$(stat -c %u:%a /var/lib/restic/stage/augaman)\" = 10001:700 && test -d /etc/restic'" + + - name: Secrets are seeded (password + repository, root 0400) + sudo: true + shell: | + set -eu + for f in /etc/restic/password /etc/restic/repository; do + test -s "$f" || { echo "missing $f: seed it from the vault (configs/restic/esh-ml1/README.md)"; exit 1; } + test "$(stat -c %U:%a "$f")" = root:400 || { echo "$f must be root 0400"; exit 1; } + done + changed_when: "false" + + - name: restic understands --repository-file + shell: restic --help | grep -q -- --repository-file + changed_when: "false" + + - name: Upload profiles.yaml + sudo: true + upload: + src: configs/restic/esh-ml1/profiles.yaml + dest: /etc/restic/profiles.yaml + mode: "0644" + + - name: Upload pre-backup.sh + sudo: true + upload: + src: configs/restic/esh-ml1/pre-backup.sh + dest: /etc/restic/pre-backup.sh + mode: "0755" + + - name: Initialise the repository (once) + sudo: true + shell: resticprofile --no-ansi --config /etc/restic/profiles.yaml --name default init + when: "! sudo -n restic --repository-file /etc/restic/repository --password-file /etc/restic/password cat config >/dev/null 2>&1" + + - name: Install the systemd timers + sudo: true + shell: resticprofile --no-ansi --config /etc/restic/profiles.yaml --name default schedule + +verify: + - name: Backup and check timers are active + shell: systemctl is-active --quiet resticprofile-backup@profile-default.timer && systemctl is-active --quiet resticprofile-check@profile-default.timer + changed_when: "false" + + # The units must EXIST before "no match" means anything: a grep over a path that + # is not there also "finds nothing". `rest:http` is the exact form the leak takes + # on the env-file hosts (seen in esh-docker-vm's unit, 2026-09-27). + - name: No repository URL (and so no rest-server password) in the generated units + shell: | + set -eu + d=/etc/systemd/system + for u in resticprofile-backup@profile-default.service resticprofile-check@profile-default.service; do + test -f "$d/$u" || { echo "missing unit $u"; exit 1; } + if grep -q 'rest:http' "$d/$u"; then echo "$u embeds the repository URL"; exit 1; fi + done + changed_when: "false" + + - name: pre-backup.sh parses + shell: bash -n /etc/restic/pre-backup.sh + changed_when: "false" diff --git a/scripts/check-backup-freshness.sh b/scripts/check-backup-freshness.sh index b8973c5..39ee726 100755 --- a/scripts/check-backup-freshness.sh +++ b/scripts/check-backup-freshness.sh @@ -160,7 +160,7 @@ discover_repos() { # $1=host $2=parent dir — prints "namenewest_epoch" done" 2>/dev/null } -EXPECTED_ANA="${EXPECTED_RESTIC_ANA:-ana-docker ana-ml2 esh-docker-vm esh-vm-db vm-esh-nas}" +EXPECTED_ANA="${EXPECTED_RESTIC_ANA:-ana-docker ana-ml2 esh-docker-vm esh-ml1 esh-vm-db vm-esh-nas}" EXPECTED_NH3="${EXPECTED_RESTIC_NH3:-irv-ml1 nh3-docker nh3-dev}" found_ana=""; found_nh3="" diff --git a/servers/esh-ml1/README.md b/servers/esh-ml1/README.md index 3a70850..9be556d 100644 --- a/servers/esh-ml1/README.md +++ b/servers/esh-ml1/README.md @@ -19,7 +19,7 @@ README. The reward seat (`/scalar-judge`) is still esh-ml1 only. | **OS** | Debian 12, unprivileged, `nesting=1,keyctl=1` | | **Size** | 6 cores, 16 GB RAM + 2 GB swap, 80 GB rootfs on `local-lvm` | | **Boot** | `onboot: 1`, `startup: order=30` — after esh-scale (1), esh-vm-db (10) and esh-vm-docker (20), so a GPU fault never delays ESH's DNS or mesh route | -| **Backups** | **None, on purpose.** esh-pve's vzdump job lists vmids explicitly and 110 is not one. Everything is rebuilt from the playbooks and the stack; models re-download. | +| **Backups** | **No vzdump, on purpose.** esh-pve's vzdump job lists vmids explicitly and 110 is not one. Everything is rebuilt from the playbooks and the stack; models re-download. **Exception: augaman's gallery**, backed up by restic daily at 0100 PT to rest-server-ana ([`configs/restic/esh-ml1/`](../../configs/restic/esh-ml1/README.md), `playbooks/esh-ml1-restic.yaml`). | ## What it serves diff --git a/stacks/augaman/.env.example b/stacks/augaman/.env.example index d407e55..16a0cb8 100644 --- a/stacks/augaman/.env.example +++ b/stacks/augaman/.env.example @@ -6,7 +6,7 @@ # compose.yaml. Do not add an env_file. # Built locally on esh-ml1 from the release tag (see README "Building"). -IMAGE=augaman:0.1.1 +IMAGE=augaman:0.1.2 PORT=8040 HOST_IP=10.0.50.80 diff --git a/stacks/augaman/README.md b/stacks/augaman/README.md index 5ae4aad..1b206f2 100644 --- a/stacks/augaman/README.md +++ b/stacks/augaman/README.md @@ -26,8 +26,13 @@ CLI, which writes `gallery.db` (mode 0600) into `BACKUP_DIR` = restore has been verified (the restored copy reports the same identities), only public-domain test fixtures may be enrolled. No household faces.** -Backup status: **NOT WIRED** as of the first deploy (2026-09-26). esh-ml1 has no -restic yet. +Backup status (2026-09-27): **WIRED AND RESTORE-VERIFIED, so the gate is met.** +restic runs daily at 0100 PT to rest-server-ana, with a fail-closed pre-backup +hook ([`configs/restic/esh-ml1/`](../../configs/restic/esh-ml1/README.md)). The +restore was verified against augaman-dev's public-domain canary identity (1 +identity, 3 samples): the copy restored from snapshot `fd3061a1` matched the +live gallery (identities, samples and embeddings, by digest), and +`integrity_check` returned ok. ```bash docker exec augaman python -m augaman.gallery.backup --db /data/gallery.db --dest /backup/gallery.db @@ -47,11 +52,17 @@ cd /opt/docker/src/augaman-vX.Y.Z && docker build -t augaman:X.Y.Z . ``` The build fetches the two pinned models and SHA-256-checks them; a mismatch fails -the build. v0.1.0 built in under 2 minutes cold; first deployed version is v0.1.1 (tightly-cropped-face detector fix). +the build. v0.1.0 built in under 2 minutes cold; v0.1.1 was the first version +deployed (2026-09-26), then v0.1.2 (2026-09-27). -⚠ **Disk:** the build took the rootfs from 66% to 80% (image ~5 GB plus build -cache). Beszel alerts at 85%. After a rebuild, remove the old image and run -`docker builder prune` once the new version is verified. +⚠ **Disk:** a build that has to install the third-party packages takes ~8–11 GB +transiently (image ~5 GB plus the ~3 GB uv download cache). Beszel alerts at 85%. +Before v0.1.2, every version bump re-installed them, and the v0.1.1 build hit 90%. +From v0.1.2 the packages install from a layer keyed on `uv export +--no-emit-project`, so a bump that changes no dependency reuses that layer. Keep +the layer cache (`docker builder prune --filter type=exec.cachemount` drops only +the download cache); a full `docker builder prune` forces the next build to +re-install everything. ## Startup is fail-closed on CUDA