feat(augaman): deploy v0.1.1 on esh-ml1:8040 (face recognition for Cicada)
Mirror pfi/augaman deploy/compose.yaml as stacks/augaman, with an .env.example and a README carrying the biometric backup gate. The image is built on esh-ml1 from a git archive of the release tag, because the box holds no gitea credentials. Serving on CUDA and visible in nvidia-smi. The gallery backup is not wired yet (esh-ml1 has no restic), so only public-domain fixtures may be enrolled. The on-box gpu test fails its batch-vs-single tolerance 3/3; reported to augaman-dev, who owns the contract.
This commit is contained in:
@@ -144,9 +144,13 @@ _As of 2026-09-26 ~1620 PT._
|
||||
|
||||
### augaman: face recognition for Cicada
|
||||
|
||||
- `pfi/augaman`; **augaman-dev is live** and has written the three contracts
|
||||
(contract review next). infra-ops' part waits for augaman-dev to reach deploy
|
||||
(see Recent decisions 2026-09-26).
|
||||
- **v0.1.1 DEPLOYED on esh-ml1:8040 2026-09-26 2347 PT** (`stacks/augaman`),
|
||||
healthy on CUDA, in `nvidia-smi`. Built on-box from a `git archive` of the tag.
|
||||
- **infra-ops owes the gallery backup NEXT:** esh-ml1 has no restic.
|
||||
`BACKUP_DIR=/var/lib/restic/stage/augaman` exists; needs restic + timer running
|
||||
the backup CLI, then one verified restore. **Fixtures only until then** (Prime).
|
||||
- `pytest -m gpu` FAILS 3/3 on the batch-vs-single tolerance (≥6e-5 vs 1e-5);
|
||||
placement passes. Reported to augaman-dev: the contract is theirs to rule on.
|
||||
|
||||
### esh-ml1
|
||||
|
||||
|
||||
@@ -42,6 +42,13 @@ because TEI cannot serve a Llama classifier. Audit and parity:
|
||||
[`stacks/reward-seat/README.md`](../../stacks/reward-seat/README.md). GPU total
|
||||
with all three: ~10.4 of 16.4 GB.
|
||||
|
||||
**Also here since 2026-09-26: augaman** — `stacks/augaman`, the face-recognition
|
||||
service for Cicada (ONNX Runtime CUDA) on `:8040`, image built on this box.
|
||||
~0.5 GB VRAM (GPU total with all four: ~11.7 of 16.4 GB). ⚠ It holds **biometric
|
||||
data** in the `augaman_gallery` volume, and this CT is outside vzdump: see the
|
||||
stack README's backup gate. ⚠ Each augaman release build needs ~10 GB of
|
||||
transient disk; prune the old image first (the v0.1.1 build hit 90%).
|
||||
|
||||
**Cut-over verified 2026-09-25** through the gateway against fv-ml1's vLLM seats
|
||||
just before they were retired: embed cosine median 0.999927 / min 0.999881
|
||||
(n=203); rerank top-1 and top-3 agreement 29/30. Two of the 30 lists contained
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
# augaman — copy to .env on the host at /opt/docker/compose/augaman/.env (mode 0600).
|
||||
# Real .env is gitignored and lives only on esh-ml1.
|
||||
#
|
||||
# ⚠ Compose's own variables carry NO `AUGAMAN_` prefix on purpose: the app refuses any
|
||||
# AUGAMAN_* variable it does not know, and compose passes only the ones named in
|
||||
# compose.yaml. Do not add an env_file.
|
||||
|
||||
# Built locally on esh-ml1 from the release tag (see README "Building").
|
||||
IMAGE=augaman:0.1.1
|
||||
|
||||
PORT=8040
|
||||
HOST_IP=10.0.50.80
|
||||
|
||||
# Where the backup CLI writes gallery.db. Owned 10001:10001 (the container user),
|
||||
# mode 0700. It is the restic stage dir, so the backup run that copies the gallery here
|
||||
# is the same run that ships it off-box (README "Backup").
|
||||
BACKUP_DIR=/var/lib/restic/stage/augaman
|
||||
|
||||
# >= 32 visible-ASCII characters. Source of truth is the vault:
|
||||
# secret get augaman/api-token
|
||||
AUGAMAN_API_TOKEN=
|
||||
@@ -0,0 +1,61 @@
|
||||
# augaman
|
||||
|
||||
**The fleet's face-recognition service for Cicada** (enroll, recognize, verify),
|
||||
on **esh-ml1** (CT 110 on esh-pve, RTX 2000E Ada). buffalo_l (SCRFD detector +
|
||||
ArcFace w600k_r50) on ONNX Runtime CUDA. Code and the canonical compose live in
|
||||
gitea `pfi/augaman` (owner: `augaman-dev`); `compose.yaml` here is a verbatim
|
||||
mirror of that repo's `deploy/compose.yaml` at the deployed tag. Change it there
|
||||
first, then re-mirror it here.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **URL** | `http://10.0.50.80:8040` (`/health` is unauthenticated; everything else needs the bearer token) |
|
||||
| **Token** | `secret get augaman/api-token` (vault is the source of truth) |
|
||||
| **Image** | `augaman:<version>`, built locally on esh-ml1 (below) |
|
||||
| **State** | named volume `augaman_gallery` (SQLite, local disk; the app refuses NFS) |
|
||||
| **VRAM** | ~1.5 GB by design (embed batches capped at 16) |
|
||||
|
||||
## ⚠ Biometric data: backup gate
|
||||
|
||||
The gallery holds face embeddings and crops of household members. esh-ml1 is
|
||||
**outside vzdump**, so the gallery reaches backup only through the app's backup
|
||||
CLI, which writes `gallery.db` (mode 0600) into `BACKUP_DIR` =
|
||||
`/var/lib/restic/stage/augaman` (owned 10001:10001, mode 0700).
|
||||
|
||||
**Operator ruling: until a scheduled backup ships that file off-box AND one
|
||||
restore has been verified (the restored copy reports the same identities), only
|
||||
public-domain test fixtures may be enrolled. No household faces.**
|
||||
|
||||
Backup status: **NOT WIRED** as of the first deploy (2026-09-26). esh-ml1 has no
|
||||
restic yet.
|
||||
|
||||
```bash
|
||||
docker exec augaman python -m augaman.gallery.backup --db /data/gallery.db --dest /backup/gallery.db
|
||||
# one JSON line + exit 0, or "backup failed: ..." + exit 1
|
||||
```
|
||||
|
||||
## Building
|
||||
|
||||
The image is built on esh-ml1 from the release tag's content. esh-ml1 has no
|
||||
gitea credentials, so the source is shipped as a `git archive`:
|
||||
|
||||
```bash
|
||||
# from nh3-dev, in a pfi/augaman checkout
|
||||
git archive --format=tar vX.Y.Z | ssh esh-ml1 'sudo mkdir -p /opt/docker/src/augaman-vX.Y.Z && sudo tar -x -C /opt/docker/src/augaman-vX.Y.Z && sudo chown -R infra-ops:infra-ops /opt/docker/src'
|
||||
# on esh-ml1
|
||||
cd /opt/docker/src/augaman-vX.Y.Z && docker build -t augaman:X.Y.Z .
|
||||
```
|
||||
|
||||
The build fetches the two pinned models and SHA-256-checks them; a mismatch fails
|
||||
the build. v0.1.0 built in under 2 minutes cold; first deployed version is v0.1.1 (tightly-cropped-face detector fix).
|
||||
|
||||
⚠ **Disk:** the build took the rootfs from 66% to 80% (image ~5 GB plus build
|
||||
cache). Beszel alerts at 85%. After a rebuild, remove the old image and run
|
||||
`docker builder prune` once the new version is verified.
|
||||
|
||||
## Startup is fail-closed on CUDA
|
||||
|
||||
The service refuses to serve unless CUDA really runs every convolution (profiled
|
||||
warmup, ORT CPU fallback disabled). `/health` reports backend `cuda` only then.
|
||||
Allow up to 180 s (`start_period`). The real GPU confirmation is the augaman
|
||||
process showing up in `nvidia-smi` on the host.
|
||||
@@ -0,0 +1,67 @@
|
||||
# augaman: the fleet's face-recognition service for Cicada (gitea pfi/augaman), on esh-ml1
|
||||
# (CT 110 on esh-pve, RTX 2000E Ada 16 GB). Enroll, recognize, verify; buffalo_l (SCRFD +
|
||||
# ArcFace w600k_r50) on ONNX Runtime CUDA. Canonical copy: this file in pfi/augaman; the
|
||||
# eshpfi stack mirrors it as stacks/augaman.
|
||||
#
|
||||
# ⚠ BIOMETRIC DATA. The gallery volume holds face embeddings and crops of household members.
|
||||
# - The live SQLite stays on the local named volume. Never NFS (the service refuses it).
|
||||
# - esh-ml1 is OUTSIDE vzdump. The gallery reaches backup only through the backup CLI,
|
||||
# writing to BACKUP_DIR, a restic-covered host path. Wire the schedule and VERIFY A
|
||||
# RESTORE before real people are enrolled (operator ruling):
|
||||
# docker exec augaman python -m augaman.gallery.backup --db /data/gallery.db --dest /backup/gallery.db
|
||||
# It prints one JSON line and exits 0, or "backup failed: ..." and exits 1.
|
||||
# - BACKUP_DIR must be writable by uid 10001, the container user. Backups are mode 0600.
|
||||
#
|
||||
# Startup refuses to serve unless CUDA really runs every convolution: a profiled warmup plus
|
||||
# ORT's CPU fallbacks turned off. /health reports backend "cuda" only then. Confirm once by the
|
||||
# process in `nvidia-smi` on the host. Warmup, including the first CUDA inference, is covered
|
||||
# by the healthcheck's start_period.
|
||||
#
|
||||
# Only AUGAMAN_* variables reach the app, and it refuses any it does not know, so compose's own
|
||||
# variables below carry no AUGAMAN_ prefix and are never passed through wholesale (no env_file).
|
||||
#
|
||||
# .env (tunables): IMAGE, PORT (8040), BACKUP_DIR, HOST_IP (10.0.50.80), AUGAMAN_API_TOKEN
|
||||
# (>= 32 visible-ASCII characters; the source of truth is the vault).
|
||||
|
||||
name: augaman
|
||||
|
||||
services:
|
||||
augaman:
|
||||
image: ${IMAGE:?set IMAGE}
|
||||
container_name: augaman
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "${PORT:-8040}:8040"
|
||||
environment:
|
||||
AUGAMAN_API_TOKEN: ${AUGAMAN_API_TOKEN:?set AUGAMAN_API_TOKEN}
|
||||
AUGAMAN_DEVICE: cuda
|
||||
AUGAMAN_CUDA_DEVICE_ID: "0"
|
||||
volumes:
|
||||
- gallery:/data
|
||||
- ${BACKUP_DIR:?set BACKUP_DIR}:/backup
|
||||
# Multipart spools of large uploads land here: RAM only, never persistent disk (api INV-A07).
|
||||
tmpfs:
|
||||
- /tmp:size=512m
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids: ["0"]
|
||||
capabilities: [gpu]
|
||||
healthcheck:
|
||||
# 200 only when ready and not degraded; a 503 (degraded) fails the check.
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8040/health', timeout=5)"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 180s
|
||||
labels:
|
||||
- homepage.group=AI - Eval & Retrieval
|
||||
- homepage.name=augaman — face recognition
|
||||
- homepage.icon=mdi-face-recognition
|
||||
- homepage.description=Enroll, recognize, verify (buffalo_l on CUDA) for Cicada
|
||||
- homepage.href=http://${HOST_IP:-10.0.50.80}:${PORT:-8040}/health
|
||||
|
||||
volumes:
|
||||
gallery:
|
||||
Reference in New Issue
Block a user