feat(augaman): v0.1.3 on esh-ml1 and fv-ml1; after-bench: GPU ~3x faster, CPU mode regressed
Both hosts are rebuilt from tag v0.1.3 (one ONNX session per detector canvas) and redeployed. pytest -m gpu tests/vision passes 3/3 on each card. Server-side for one face, same harness as the v0.1.2 baseline: - esh-ml1 GPU 144 -> 48 ms - fv-ml1 GPU 75 -> 27 ms End to end from nh3-dev: 101.6 and 73.8 ms. CPU mode got slower on every CPU target: fv-ml1 cpuset 0-5 went 152 -> 205 ms with a face, and the no-face frame roughly doubled. That is well outside the run-to-run spread. The suspected cause (not measured) is per-session ORT thread pools spinning. Reported to augaman-dev. Neither deployment uses CPU mode. On esh-ml1 the dependency layer missed the build cache and the rootfs touched 90% until the v0.1.2 image was removed. fv-ml1's build hit the cache, and the exported requirements are identical, so the stack README now says to check disk before building on esh-ml1.
This commit is contained in:
@@ -6,7 +6,7 @@
|
||||
# compose.yaml. Do not add an env_file.
|
||||
|
||||
# Built locally on esh-ml1 from the release tag (see README "Building").
|
||||
IMAGE=augaman:0.1.2
|
||||
IMAGE=augaman:0.1.3
|
||||
|
||||
PORT=8040
|
||||
HOST_IP=10.0.50.80
|
||||
|
||||
@@ -60,7 +60,8 @@ cd /opt/docker/src/augaman-vX.Y.Z && docker build -t augaman:X.Y.Z .
|
||||
|
||||
The build fetches the two pinned models and SHA-256-checks them; a mismatch fails
|
||||
the build. v0.1.0 built in under 2 minutes cold; v0.1.1 was the first version
|
||||
deployed (2026-09-26), then v0.1.2 (2026-09-27).
|
||||
deployed (2026-09-26), then v0.1.2 and v0.1.3 (2026-09-27; v0.1.3 gives each
|
||||
detector canvas its own session, about +190 MiB VRAM on esh-ml1).
|
||||
|
||||
⚠ **Disk:** a build that has to install the third-party packages takes ~8–11 GB
|
||||
transiently (image ~5 GB plus the ~3 GB uv download cache). Beszel alerts at 85%.
|
||||
@@ -69,7 +70,13 @@ From v0.1.2 the packages install from a layer keyed on `uv export
|
||||
--no-emit-project`, so a bump that changes no dependency reuses that layer. Keep
|
||||
the layer cache (`docker builder prune --filter type=exec.cachemount` drops only
|
||||
the download cache); a full `docker builder prune` forces the next build to
|
||||
re-install everything.
|
||||
re-install everything. ⚠ **The cache did not survive on esh-ml1 for v0.1.3** (the
|
||||
deps layer re-ran for 1m43s, and the rootfs touched 90% until the old image was
|
||||
removed). On fv-ml1 the same build was CACHED (49 s), and the exported requirements
|
||||
were byte-identical across the two versions, so the layering itself works. The
|
||||
esh-ml1 miss is unexplained; suspect the cache pruning done there after v0.1.2.
|
||||
**On esh-ml1, check `df` before a build, and have ≥15 GB free or remove the old
|
||||
image first.**
|
||||
|
||||
## Startup is fail-closed on CUDA
|
||||
|
||||
|
||||
Reference in New Issue
Block a user