# augaman

**The fleet's face-recognition service for Cicada** (enroll, recognize, verify),
on **esh-ml1** (CT 110 on esh-pve, RTX 2000E Ada). buffalo_l (SCRFD detector +
ArcFace w600k_r50) on ONNX Runtime CUDA. Code and the canonical compose live in
gitea `pfi/augaman` (owner: `augaman-dev`); `compose.yaml` here is a verbatim
mirror of that repo's `deploy/compose.yaml` at the deployed tag. Change it there
first, then re-mirror it here.

| | |
|---|---|
| **URL** | `http://10.0.50.80:8040` (`/health` is unauthenticated; everything else needs the bearer token) |
| **Token** | `secret get augaman/api-token` (vault is the source of truth) |
| **Image** | `augaman:<version>`, built locally on esh-ml1 (below) |
| **State** | named volume `augaman_gallery` (SQLite, local disk; the app refuses NFS) |
| **VRAM** | ~0.74 GB on v0.1.3 (measured after the bench; 0.51 GB on v0.1.2) |

**One instance: esh-ml1.** A second, fixtures-only instance ran on fv-ml1's GPU 1
(`GPU_ID=1`) from 2026-09-27 0017 to ~0130 PT, for the speed comparison in
[`docs/pfi/augaman-speed-bench/`](../../docs/pfi/augaman-speed-bench/README.md).
**Prime removed it** after v0.1.3 put esh-ml1 at ~48 ms per face: it is in the house
next to the cameras and it holds the verified backup. `GPU_ID` / `CARD_SUFFIX` stay in
the compose for any future second host.

## ⚠ Biometric data: backup gate

The gallery holds face embeddings and crops of household members. esh-ml1 is
**outside vzdump**, so the gallery reaches backup only through the app's backup
CLI, which writes `gallery.db` (mode 0600) into `BACKUP_DIR` =
`/var/lib/restic/stage/augaman` (owned 10001:10001, mode 0700).

**Operator ruling: until a scheduled backup ships that file off-box AND one
restore has been verified (the restored copy reports the same identities), only
public-domain test fixtures may be enrolled. No household faces.**

Backup status (2026-09-27): **WIRED AND RESTORE-VERIFIED, so the gate is met.**
restic runs daily at 0100 PT to rest-server-ana, with a fail-closed pre-backup
hook ([`configs/restic/esh-ml1/`](../../configs/restic/esh-ml1/README.md)). The
restore was verified against augaman-dev's public-domain canary identity (1
identity, 3 samples): the copy restored from snapshot `fd3061a1` matched the
live gallery (identities, samples and embeddings, by digest), and
`integrity_check` returned ok.

```bash
docker exec augaman python -m augaman.gallery.backup --db /data/gallery.db --dest /backup/gallery.db
# one JSON line + exit 0, or "backup failed: ..." + exit 1
```

## Building

The image is built on esh-ml1 from the release tag's content. esh-ml1 has no
gitea credentials, so the source is shipped as a `git archive`:

```bash
# from nh3-dev, in a pfi/augaman checkout
git archive --format=tar vX.Y.Z | ssh esh-ml1 'sudo mkdir -p /opt/docker/src/augaman-vX.Y.Z && sudo tar -x -C /opt/docker/src/augaman-vX.Y.Z && sudo chown -R infra-ops:infra-ops /opt/docker/src'
# on esh-ml1
cd /opt/docker/src/augaman-vX.Y.Z && docker build -t augaman:X.Y.Z .
```

The build fetches the two pinned models and SHA-256-checks them; a mismatch fails
the build. v0.1.0 built in under 2 minutes cold; v0.1.1 was the first version
deployed (2026-09-26), then v0.1.2 and v0.1.3 (2026-09-27; v0.1.3 gives each
detector canvas its own session, about +190 MiB VRAM on esh-ml1).

⚠ **Disk:** a build that has to install the third-party packages takes ~8–11 GB
transiently (image ~5 GB plus the ~3 GB uv download cache). Beszel alerts at 85%.
Before v0.1.2, every version bump re-installed them, and the v0.1.1 build hit 90%.
From v0.1.2 the packages install from a layer keyed on `uv export
--no-emit-project`, so a bump that changes no dependency reuses that layer. Keep
the layer cache (`docker builder prune --filter type=exec.cachemount` drops only
the download cache); a full `docker builder prune` forces the next build to
re-install everything. ⚠ **The cache did not survive on esh-ml1 for v0.1.3** (the
deps layer re-ran for 1m43s, and the rootfs touched 90% until the old image was
removed). On fv-ml1 the same build was CACHED (49 s), and the exported requirements
were byte-identical across the two versions, so the layering itself works. The
esh-ml1 miss is unexplained; suspect the cache pruning done there after v0.1.2.
**On esh-ml1, check `df` before a build, and have ≥15 GB free or remove the old
image first.**

## Startup is fail-closed on CUDA

The service refuses to serve unless CUDA really runs every convolution (profiled
warmup, ORT CPU fallback disabled). `/health` reports backend `cuda` only then.
Allow up to 180 s (`start_period`). The real GPU confirmation is the augaman
process showing up in `nvidia-smi` on the host.