diff --git a/persistent-memory.md b/persistent-memory.md index e41a07d..faac2ef 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -144,9 +144,13 @@ _As of 2026-09-26 ~1620 PT._ ### augaman: face recognition for Cicada -- `pfi/augaman`; **augaman-dev is live** and has written the three contracts - (contract review next). infra-ops' part waits for augaman-dev to reach deploy - (see Recent decisions 2026-09-26). +- **v0.1.1 DEPLOYED on esh-ml1:8040 2026-09-26 2347 PT** (`stacks/augaman`), + healthy on CUDA, in `nvidia-smi`. Built on-box from a `git archive` of the tag. +- **infra-ops owes the gallery backup NEXT:** esh-ml1 has no restic. + `BACKUP_DIR=/var/lib/restic/stage/augaman` exists; needs restic + timer running + the backup CLI, then one verified restore. **Fixtures only until then** (Prime). +- `pytest -m gpu` FAILS 3/3 on the batch-vs-single tolerance (≥6e-5 vs 1e-5); + placement passes. Reported to augaman-dev: the contract is theirs to rule on. ### esh-ml1 diff --git a/servers/esh-ml1/README.md b/servers/esh-ml1/README.md index 887e153..3a70850 100644 --- a/servers/esh-ml1/README.md +++ b/servers/esh-ml1/README.md @@ -42,6 +42,13 @@ because TEI cannot serve a Llama classifier. Audit and parity: [`stacks/reward-seat/README.md`](../../stacks/reward-seat/README.md). GPU total with all three: ~10.4 of 16.4 GB. +**Also here since 2026-09-26: augaman** — `stacks/augaman`, the face-recognition +service for Cicada (ONNX Runtime CUDA) on `:8040`, image built on this box. +~0.5 GB VRAM (GPU total with all four: ~11.7 of 16.4 GB). ⚠ It holds **biometric +data** in the `augaman_gallery` volume, and this CT is outside vzdump: see the +stack README's backup gate. ⚠ Each augaman release build needs ~10 GB of +transient disk; prune the old image first (the v0.1.1 build hit 90%). + **Cut-over verified 2026-09-25** through the gateway against fv-ml1's vLLM seats just before they were retired: embed cosine median 0.999927 / min 0.999881 (n=203); rerank top-1 and top-3 agreement 29/30. Two of the 30 lists contained diff --git a/stacks/augaman/.env.example b/stacks/augaman/.env.example new file mode 100644 index 0000000..d407e55 --- /dev/null +++ b/stacks/augaman/.env.example @@ -0,0 +1,21 @@ +# augaman — copy to .env on the host at /opt/docker/compose/augaman/.env (mode 0600). +# Real .env is gitignored and lives only on esh-ml1. +# +# ⚠ Compose's own variables carry NO `AUGAMAN_` prefix on purpose: the app refuses any +# AUGAMAN_* variable it does not know, and compose passes only the ones named in +# compose.yaml. Do not add an env_file. + +# Built locally on esh-ml1 from the release tag (see README "Building"). +IMAGE=augaman:0.1.1 + +PORT=8040 +HOST_IP=10.0.50.80 + +# Where the backup CLI writes gallery.db. Owned 10001:10001 (the container user), +# mode 0700. It is the restic stage dir, so the backup run that copies the gallery here +# is the same run that ships it off-box (README "Backup"). +BACKUP_DIR=/var/lib/restic/stage/augaman + +# >= 32 visible-ASCII characters. Source of truth is the vault: +# secret get augaman/api-token +AUGAMAN_API_TOKEN= diff --git a/stacks/augaman/README.md b/stacks/augaman/README.md new file mode 100644 index 0000000..5ae4aad --- /dev/null +++ b/stacks/augaman/README.md @@ -0,0 +1,61 @@ +# augaman + +**The fleet's face-recognition service for Cicada** (enroll, recognize, verify), +on **esh-ml1** (CT 110 on esh-pve, RTX 2000E Ada). buffalo_l (SCRFD detector + +ArcFace w600k_r50) on ONNX Runtime CUDA. Code and the canonical compose live in +gitea `pfi/augaman` (owner: `augaman-dev`); `compose.yaml` here is a verbatim +mirror of that repo's `deploy/compose.yaml` at the deployed tag. Change it there +first, then re-mirror it here. + +| | | +|---|---| +| **URL** | `http://10.0.50.80:8040` (`/health` is unauthenticated; everything else needs the bearer token) | +| **Token** | `secret get augaman/api-token` (vault is the source of truth) | +| **Image** | `augaman:`, built locally on esh-ml1 (below) | +| **State** | named volume `augaman_gallery` (SQLite, local disk; the app refuses NFS) | +| **VRAM** | ~1.5 GB by design (embed batches capped at 16) | + +## ⚠ Biometric data: backup gate + +The gallery holds face embeddings and crops of household members. esh-ml1 is +**outside vzdump**, so the gallery reaches backup only through the app's backup +CLI, which writes `gallery.db` (mode 0600) into `BACKUP_DIR` = +`/var/lib/restic/stage/augaman` (owned 10001:10001, mode 0700). + +**Operator ruling: until a scheduled backup ships that file off-box AND one +restore has been verified (the restored copy reports the same identities), only +public-domain test fixtures may be enrolled. No household faces.** + +Backup status: **NOT WIRED** as of the first deploy (2026-09-26). esh-ml1 has no +restic yet. + +```bash +docker exec augaman python -m augaman.gallery.backup --db /data/gallery.db --dest /backup/gallery.db +# one JSON line + exit 0, or "backup failed: ..." + exit 1 +``` + +## Building + +The image is built on esh-ml1 from the release tag's content. esh-ml1 has no +gitea credentials, so the source is shipped as a `git archive`: + +```bash +# from nh3-dev, in a pfi/augaman checkout +git archive --format=tar vX.Y.Z | ssh esh-ml1 'sudo mkdir -p /opt/docker/src/augaman-vX.Y.Z && sudo tar -x -C /opt/docker/src/augaman-vX.Y.Z && sudo chown -R infra-ops:infra-ops /opt/docker/src' +# on esh-ml1 +cd /opt/docker/src/augaman-vX.Y.Z && docker build -t augaman:X.Y.Z . +``` + +The build fetches the two pinned models and SHA-256-checks them; a mismatch fails +the build. v0.1.0 built in under 2 minutes cold; first deployed version is v0.1.1 (tightly-cropped-face detector fix). + +⚠ **Disk:** the build took the rootfs from 66% to 80% (image ~5 GB plus build +cache). Beszel alerts at 85%. After a rebuild, remove the old image and run +`docker builder prune` once the new version is verified. + +## Startup is fail-closed on CUDA + +The service refuses to serve unless CUDA really runs every convolution (profiled +warmup, ORT CPU fallback disabled). `/health` reports backend `cuda` only then. +Allow up to 180 s (`start_period`). The real GPU confirmation is the augaman +process showing up in `nvidia-smi` on the host. diff --git a/stacks/augaman/compose.yaml b/stacks/augaman/compose.yaml new file mode 100644 index 0000000..23fcd58 --- /dev/null +++ b/stacks/augaman/compose.yaml @@ -0,0 +1,67 @@ +# augaman: the fleet's face-recognition service for Cicada (gitea pfi/augaman), on esh-ml1 +# (CT 110 on esh-pve, RTX 2000E Ada 16 GB). Enroll, recognize, verify; buffalo_l (SCRFD + +# ArcFace w600k_r50) on ONNX Runtime CUDA. Canonical copy: this file in pfi/augaman; the +# eshpfi stack mirrors it as stacks/augaman. +# +# ⚠ BIOMETRIC DATA. The gallery volume holds face embeddings and crops of household members. +# - The live SQLite stays on the local named volume. Never NFS (the service refuses it). +# - esh-ml1 is OUTSIDE vzdump. The gallery reaches backup only through the backup CLI, +# writing to BACKUP_DIR, a restic-covered host path. Wire the schedule and VERIFY A +# RESTORE before real people are enrolled (operator ruling): +# docker exec augaman python -m augaman.gallery.backup --db /data/gallery.db --dest /backup/gallery.db +# It prints one JSON line and exits 0, or "backup failed: ..." and exits 1. +# - BACKUP_DIR must be writable by uid 10001, the container user. Backups are mode 0600. +# +# Startup refuses to serve unless CUDA really runs every convolution: a profiled warmup plus +# ORT's CPU fallbacks turned off. /health reports backend "cuda" only then. Confirm once by the +# process in `nvidia-smi` on the host. Warmup, including the first CUDA inference, is covered +# by the healthcheck's start_period. +# +# Only AUGAMAN_* variables reach the app, and it refuses any it does not know, so compose's own +# variables below carry no AUGAMAN_ prefix and are never passed through wholesale (no env_file). +# +# .env (tunables): IMAGE, PORT (8040), BACKUP_DIR, HOST_IP (10.0.50.80), AUGAMAN_API_TOKEN +# (>= 32 visible-ASCII characters; the source of truth is the vault). + +name: augaman + +services: + augaman: + image: ${IMAGE:?set IMAGE} + container_name: augaman + restart: unless-stopped + ports: + - "${PORT:-8040}:8040" + environment: + AUGAMAN_API_TOKEN: ${AUGAMAN_API_TOKEN:?set AUGAMAN_API_TOKEN} + AUGAMAN_DEVICE: cuda + AUGAMAN_CUDA_DEVICE_ID: "0" + volumes: + - gallery:/data + - ${BACKUP_DIR:?set BACKUP_DIR}:/backup + # Multipart spools of large uploads land here: RAM only, never persistent disk (api INV-A07). + tmpfs: + - /tmp:size=512m + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: ["0"] + capabilities: [gpu] + healthcheck: + # 200 only when ready and not degraded; a 503 (degraded) fails the check. + test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8040/health', timeout=5)"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 180s + labels: + - homepage.group=AI - Eval & Retrieval + - homepage.name=augaman — face recognition + - homepage.icon=mdi-face-recognition + - homepage.description=Enroll, recognize, verify (buffalo_l on CUDA) for Cicada + - homepage.href=http://${HOST_IP:-10.0.50.80}:${PORT:-8040}/health + +volumes: + gallery: