From ef64a69e303ccf272f97d95dabfbaad66bd5be54 Mon Sep 17 00:00:00 2001
From: Vuong Hoang <vh@phasefinal.com>
Date: Sun, 27 Sep 2026 01:13:01 -0700
Subject: [PATCH] feat(augaman): v0.1.3 on esh-ml1 and fv-ml1; after-bench: GPU
 ~3x faster, CPU mode regressed

Both hosts are rebuilt from tag v0.1.3 (one ONNX session per detector canvas) and
redeployed. pytest -m gpu tests/vision passes 3/3 on each card.

Server-side for one face, same harness as the v0.1.2 baseline:
- esh-ml1 GPU 144 -> 48 ms
- fv-ml1 GPU 75 -> 27 ms
End to end from nh3-dev: 101.6 and 73.8 ms.

CPU mode got slower on every CPU target: fv-ml1 cpuset 0-5 went 152 -> 205 ms
with a face, and the no-face frame roughly doubled. That is well outside the
run-to-run spread. The suspected cause (not measured) is per-session ORT
thread pools spinning. Reported to augaman-dev. Neither deployment uses CPU
mode.

On esh-ml1 the dependency layer missed the build cache and the rootfs touched
90% until the v0.1.2 image was removed. fv-ml1's build hit the cache, and the
exported requirements are identical, so the stack README now says to check
disk before building on esh-ml1.
---
 docs/pfi/augaman-speed-bench/README.md        |  40 +-
 .../rows-2026-09-27-v0.1.3.json               | 542 ++++++++++++++++++
 stacks/augaman/.env.example                   |   2 +-
 stacks/augaman/README.md                      |  11 +-
 4 files changed, 589 insertions(+), 6 deletions(-)
 create mode 100644 docs/pfi/augaman-speed-bench/rows-2026-09-27-v0.1.3.json

diff --git a/docs/pfi/augaman-speed-bench/README.md b/docs/pfi/augaman-speed-bench/README.md
index 8c13b13..b49d572 100644
--- a/docs/pfi/augaman-speed-bench/README.md
+++ b/docs/pfi/augaman-speed-bench/README.md
@@ -1,8 +1,42 @@
 # augaman speed bench: CPU vs esh-ml1 GPU vs fv-ml1 GPU
 
-Asked by Prime on 2026-09-27. This is the **v0.1.2 "before"** baseline. v0.1.3 gives
-each detector canvas its own ONNX session, removing the ~90 ms CUDA shape-switching
-cost that comfy-dev found. The same harness is to be re-run on v0.1.3 for the "after".
+Asked by Prime on 2026-09-27. Two passes of the same harness:
+**v0.1.2 (before, 0021–0026 PT)** and **v0.1.3 (after, 0106–0112 PT)**. v0.1.3 gives
+each detector canvas its own ONNX session, which removes the ~90 ms CUDA
+shape-switching cost that comfy-dev found.
+
+## Headline: server-side ms per frame, one face (a) / no face (c)
+
+| target | v0.1.2 | **v0.1.3** | change |
+|---|---|---|---|
+| esh-ml1 GPU (RTX 2000E) | 143.8 / 136.2 | **48.0 / 37.2** | ~3.0× faster |
+| fv-ml1 GPU1 (RTX PRO 6000) | 75.0 / 69.5 | **27.3 / 22.9** | ~2.8× faster |
+| fv-ml1 CPU, cpuset 0-5 | 152.4 / 72.3 | **205.1 / 133.2** | ⚠ ~1.35× / 1.8× SLOWER |
+| fv-ml1 CPU, cpuset 0-23 | 155.9 / 61.9 | **204.6 / 124.0** | ⚠ SLOWER |
+| esh-ml1 CPU, 6 LXC threads | 887.6 / 159.4 | **1105.1 / 395.4** | ⚠ SLOWER |
+
+End to end from nh3-dev, frame (a) p50: esh-ml1 GPU **101.6 ms** (was 196.5) and fv-ml1
+GPU **73.8 ms** (was 126.6). The /health floor is ~27 ms for both.
+
+**The GPU gain is real, and CPU mode regressed.** The CPU slowdown sits far outside the
+run-to-run spread: fv CPU6 (c) ran 158–166 ms against 111–113 before. It shows on
+every CPU target and hits the no-face frame hardest, which points at the detector
+sessions. Hypothesis, NOT measured: each of the three ORT sessions now has its own
+intra-op thread pool, whose threads spin while another session runs, so the CPU
+oversubscribes. The candidate fixes are `session.intra_op.allow_spinning=0` or a shared
+global thread pool. Reported to augaman-dev. Neither deployment runs in CPU mode, so
+nothing live is affected.
+
+VRAM after the bench (nvidia-smi): esh-ml1 736 MiB (514 on v0.1.2), fv-ml1 1828 MiB
+(1264 on v0.1.2). `pytest -m gpu tests/vision` passes 3/3 on both cards on v0.1.3
+(three sessions proven: detector@128, detector@640, embedder). Raw data:
+`rows-2026-09-27-v0.1.3.json`, and GPU utilisation samples every ~15 s in
+`gpu-util-2026-09-27-v0.1.3.log` (mostly 0%, with peaks of 21% on esh-ml1 and 31%
+on fv-ml1 GPU 1; fv-ml1 GPU 2 hit 99% once from another seat's traffic).
+
+---
+
+# v0.1.2 baseline detail
 
 ## Harness (it is part of the number)
 
diff --git a/docs/pfi/augaman-speed-bench/rows-2026-09-27-v0.1.3.json b/docs/pfi/augaman-speed-bench/rows-2026-09-27-v0.1.3.json
new file mode 100644
index 0000000..d3a0ef7
--- /dev/null
+++ b/docs/pfi/augaman-speed-bench/rows-2026-09-27-v0.1.3.json
@@ -0,0 +1,542 @@
+[
+ {
+  "target": "esh-ml1 GPU",
+  "frame": "health_floor",
+  "run": 1,
+  "p50": 28.56831700046314,
+  "p90": 35.426961704797584,
+  "server_probe_mean": null,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 GPU",
+  "frame": "a_1080p_1face",
+  "run": 1,
+  "p50": 103.24051550560398,
+  "p90": 108.98464530328056,
+  "server_probe_mean": 50.50277935079066,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 GPU",
+  "frame": "b_960x1001_2face",
+  "run": 1,
+  "p50": 91.28140949906083,
+  "p90": 97.65329169604229,
+  "server_probe_mean": 36.06587980029872,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 GPU",
+  "frame": "c_1080p_0face",
+  "run": 1,
+  "p50": 80.58081250055693,
+  "p90": 88.59687689837301,
+  "server_probe_mean": 37.25972330139484,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 GPU1",
+  "frame": "health_floor",
+  "run": 1,
+  "p50": 31.64928349724505,
+  "p90": 33.86286070744973,
+  "server_probe_mean": null,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 GPU1",
+  "frame": "a_1080p_1face",
+  "run": 1,
+  "p50": 77.40206849848619,
+  "p90": 87.3496351967333,
+  "server_probe_mean": 26.340403244830668,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 GPU1",
+  "frame": "b_960x1001_2face",
+  "run": 1,
+  "p50": 73.34154599084286,
+  "p90": 77.04311340639833,
+  "server_probe_mean": 24.761526938527822,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 GPU1",
+  "frame": "c_1080p_0face",
+  "run": 1,
+  "p50": 60.86771749687614,
+  "p90": 64.20872810704168,
+  "server_probe_mean": 22.897095198277384,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 CPU6",
+  "frame": "health_floor",
+  "run": 1,
+  "p50": 31.80485299526481,
+  "p90": 37.67810769495554,
+  "server_probe_mean": null,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 CPU6",
+  "frame": "a_1080p_1face",
+  "run": 1,
+  "p50": 1157.2700699980487,
+  "p90": 1285.2505169998042,
+  "server_probe_mean": 1105.1321487975656,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 CPU6",
+  "frame": "b_960x1001_2face",
+  "run": 1,
+  "p50": 1620.7859505011584,
+  "p90": 1897.2889195065363,
+  "server_probe_mean": 1567.7373465063283,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 CPU6",
+  "frame": "c_1080p_0face",
+  "run": 1,
+  "p50": 358.5786059993552,
+  "p90": 447.7066722058226,
+  "server_probe_mean": 325.5370168029913,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU6",
+  "frame": "health_floor",
+  "run": 1,
+  "p50": 29.225862497696653,
+  "p90": 31.134746607858688,
+  "server_probe_mean": null,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU6",
+  "frame": "a_1080p_1face",
+  "run": 1,
+  "p50": 244.82941399764968,
+  "p90": 285.62337980692973,
+  "server_probe_mean": 205.05198980681598,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU6",
+  "frame": "b_960x1001_2face",
+  "run": 1,
+  "p50": 273.4833980066469,
+  "p90": 318.48532019648707,
+  "server_probe_mean": 225.52827950567007,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU6",
+  "frame": "c_1080p_0face",
+  "run": 1,
+  "p50": 165.87754249485442,
+  "p90": 204.7811397977057,
+  "server_probe_mean": 133.18235542392358,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU24",
+  "frame": "health_floor",
+  "run": 1,
+  "p50": 28.081258496968076,
+  "p90": 32.582762498350355,
+  "server_probe_mean": null,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU24",
+  "frame": "a_1080p_1face",
+  "run": 1,
+  "p50": 247.0757969931583,
+  "p90": 265.46713550778804,
+  "server_probe_mean": 200.1803957624361,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU24",
+  "frame": "b_960x1001_2face",
+  "run": 1,
+  "p50": 273.32474349532276,
+  "p90": 343.1363677998889,
+  "server_probe_mean": 240.7834232551977,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU24",
+  "frame": "c_1080p_0face",
+  "run": 1,
+  "p50": 166.3104715044028,
+  "p90": 182.07621549954638,
+  "server_probe_mean": 124.0462520509027,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 GPU",
+  "frame": "health_floor",
+  "run": 2,
+  "p50": 26.54249199986225,
+  "p90": 27.6983294999809,
+  "server_probe_mean": null,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 GPU",
+  "frame": "a_1080p_1face",
+  "run": 2,
+  "p50": 101.5832139964914,
+  "p90": 109.36262830364285,
+  "server_probe_mean": 47.99011784780305,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 GPU",
+  "frame": "b_960x1001_2face",
+  "run": 2,
+  "p50": 94.75307750108186,
+  "p90": 100.28928269312019,
+  "server_probe_mean": 36.026441550347954,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 GPU",
+  "frame": "c_1080p_0face",
+  "run": 2,
+  "p50": 78.17531850014348,
+  "p90": 83.03192700113868,
+  "server_probe_mean": 37.16747584985569,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 GPU1",
+  "frame": "health_floor",
+  "run": 2,
+  "p50": 26.91060849610949,
+  "p90": 27.50819110951852,
+  "server_probe_mean": null,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 GPU1",
+  "frame": "a_1080p_1face",
+  "run": 2,
+  "p50": 73.84634450136218,
+  "p90": 75.3575598981115,
+  "server_probe_mean": 27.809194929432124,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 GPU1",
+  "frame": "b_960x1001_2face",
+  "run": 2,
+  "p50": 73.40573299734388,
+  "p90": 76.6928782017203,
+  "server_probe_mean": 25.148114131297916,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 GPU1",
+  "frame": "c_1080p_0face",
+  "run": 2,
+  "p50": 60.866275503940415,
+  "p90": 64.77840160223423,
+  "server_probe_mean": 21.837973268702626,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 CPU6",
+  "frame": "health_floor",
+  "run": 2,
+  "p50": 30.955315494793467,
+  "p90": 34.70198850845919,
+  "server_probe_mean": null,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 CPU6",
+  "frame": "a_1080p_1face",
+  "run": 2,
+  "p50": 1048.8297365009203,
+  "p90": 1299.0026480983945,
+  "server_probe_mean": 1022.0766173981247,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 CPU6",
+  "frame": "b_960x1001_2face",
+  "run": 2,
+  "p50": 1584.0665085052024,
+  "p90": 1715.8558561946847,
+  "server_probe_mean": 1513.3129546549753,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 CPU6",
+  "frame": "c_1080p_0face",
+  "run": 2,
+  "p50": 433.8740794992191,
+  "p90": 538.4446465948713,
+  "server_probe_mean": 395.367383849225,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU6",
+  "frame": "health_floor",
+  "run": 2,
+  "p50": 29.1177155013429,
+  "p90": 31.4779651016579,
+  "server_probe_mean": null,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU6",
+  "frame": "a_1080p_1face",
+  "run": 2,
+  "p50": 258.8035505032167,
+  "p90": 279.1849860892398,
+  "server_probe_mean": 204.92596072144806,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU6",
+  "frame": "b_960x1001_2face",
+  "run": 2,
+  "p50": 253.7849774962524,
+  "p90": 302.2328945007757,
+  "server_probe_mean": 216.03364354232326,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU6",
+  "frame": "c_1080p_0face",
+  "run": 2,
+  "p50": 159.76047250296688,
+  "p90": 347.6618904911466,
+  "server_probe_mean": 378.62427223008126,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU24",
+  "frame": "health_floor",
+  "run": 2,
+  "p50": 28.286710505199153,
+  "p90": 33.18228400457884,
+  "server_probe_mean": null,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU24",
+  "frame": "a_1080p_1face",
+  "run": 2,
+  "p50": 250.35845449747285,
+  "p90": 270.55101390578784,
+  "server_probe_mean": 204.57896552979946,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU24",
+  "frame": "b_960x1001_2face",
+  "run": 2,
+  "p50": 276.2931550023495,
+  "p90": 302.7462427984574,
+  "server_probe_mean": 228.16327152540907,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU24",
+  "frame": "c_1080p_0face",
+  "run": 2,
+  "p50": 173.82631749933353,
+  "p90": 203.72216301038867,
+  "server_probe_mean": 144.8986719478853,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 GPU",
+  "frame": "health_floor",
+  "run": 3,
+  "p50": 26.987262994225603,
+  "p90": 30.949878702813294,
+  "server_probe_mean": null,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 GPU",
+  "frame": "a_1080p_1face",
+  "run": 3,
+  "p50": 99.26728250138694,
+  "p90": 105.58418770378921,
+  "server_probe_mean": 45.727238450490404,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 GPU",
+  "frame": "b_960x1001_2face",
+  "run": 3,
+  "p50": 92.11020849761553,
+  "p90": 96.73994110344212,
+  "server_probe_mean": 34.403727897733916,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 GPU",
+  "frame": "c_1080p_0face",
+  "run": 3,
+  "p50": 79.78740399994422,
+  "p90": 85.02629639551742,
+  "server_probe_mean": 37.2430077943136,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 GPU1",
+  "frame": "health_floor",
+  "run": 3,
+  "p50": 27.467196501675062,
+  "p90": 32.5130715049454,
+  "server_probe_mean": null,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 GPU1",
+  "frame": "a_1080p_1face",
+  "run": 3,
+  "p50": 73.05359399470035,
+  "p90": 79.36581129324622,
+  "server_probe_mean": 27.340552653186023,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 GPU1",
+  "frame": "b_960x1001_2face",
+  "run": 3,
+  "p50": 72.0015334954951,
+  "p90": 75.61690630245721,
+  "server_probe_mean": 24.976778693962842,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 GPU1",
+  "frame": "c_1080p_0face",
+  "run": 3,
+  "p50": 61.28011599503225,
+  "p90": 63.8927058011177,
+  "server_probe_mean": 22.860179003328085,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 CPU6",
+  "frame": "health_floor",
+  "run": 3,
+  "p50": 28.69356150040403,
+  "p90": 30.96141718851868,
+  "server_probe_mean": null,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 CPU6",
+  "frame": "a_1080p_1face",
+  "run": 3,
+  "p50": 1115.432997001335,
+  "p90": 1451.1805613117758,
+  "server_probe_mean": 1106.370482649072,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 CPU6",
+  "frame": "b_960x1001_2face",
+  "run": 3,
+  "p50": 1618.1769984978018,
+  "p90": 1854.3222563093877,
+  "server_probe_mean": 1573.5961115453392,
+  "bad": 0
+ },
+ {
+  "target": "esh-ml1 CPU6",
+  "frame": "c_1080p_0face",
+  "run": 3,
+  "p50": 449.7984624977107,
+  "p90": 541.9355103018461,
+  "server_probe_mean": 404.3181982007809,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU6",
+  "frame": "health_floor",
+  "run": 3,
+  "p50": 29.24010700371582,
+  "p90": 33.91167670779396,
+  "server_probe_mean": null,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU6",
+  "frame": "a_1080p_1face",
+  "run": 3,
+  "p50": 248.83804350247374,
+  "p90": 305.8311790940934,
+  "server_probe_mean": 207.7646869351156,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU6",
+  "frame": "b_960x1001_2face",
+  "run": 3,
+  "p50": 252.01708450185833,
+  "p90": 323.74196870077867,
+  "server_probe_mean": 215.00399404903874,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU6",
+  "frame": "c_1080p_0face",
+  "run": 3,
+  "p50": 158.14820949890418,
+  "p90": 203.81181769480463,
+  "server_probe_mean": 131.31015392718837,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU24",
+  "frame": "health_floor",
+  "run": 3,
+  "p50": 29.235157504444942,
+  "p90": 33.62159581010929,
+  "server_probe_mean": null,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU24",
+  "frame": "a_1080p_1face",
+  "run": 3,
+  "p50": 251.2742774997605,
+  "p90": 272.51768180867657,
+  "server_probe_mean": 206.39763702638447,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU24",
+  "frame": "b_960x1001_2face",
+  "run": 3,
+  "p50": 265.6091139942873,
+  "p90": 297.4889616059954,
+  "server_probe_mean": 222.6791250752285,
+  "bad": 0
+ },
+ {
+  "target": "fv-ml1 CPU24",
+  "frame": "c_1080p_0face",
+  "run": 3,
+  "p50": 158.92017900478095,
+  "p90": 181.67336140904808,
+  "server_probe_mean": 121.74809173448011,
+  "bad": 0
+ }
+]
\ No newline at end of file
diff --git a/stacks/augaman/.env.example b/stacks/augaman/.env.example
index ff9dc61..2e29582 100644
--- a/stacks/augaman/.env.example
+++ b/stacks/augaman/.env.example
@@ -6,7 +6,7 @@
 # compose.yaml. Do not add an env_file.
 
 # Built locally on esh-ml1 from the release tag (see README "Building").
-IMAGE=augaman:0.1.2
+IMAGE=augaman:0.1.3
 
 PORT=8040
 HOST_IP=10.0.50.80
diff --git a/stacks/augaman/README.md b/stacks/augaman/README.md
index 8ccdeb3..7e9ddc0 100644
--- a/stacks/augaman/README.md
+++ b/stacks/augaman/README.md
@@ -60,7 +60,8 @@ cd /opt/docker/src/augaman-vX.Y.Z && docker build -t augaman:X.Y.Z .
 
 The build fetches the two pinned models and SHA-256-checks them; a mismatch fails
 the build. v0.1.0 built in under 2 minutes cold; v0.1.1 was the first version
-deployed (2026-09-26), then v0.1.2 (2026-09-27).
+deployed (2026-09-26), then v0.1.2 and v0.1.3 (2026-09-27; v0.1.3 gives each
+detector canvas its own session, about +190 MiB VRAM on esh-ml1).
 
 ⚠ **Disk:** a build that has to install the third-party packages takes ~8–11 GB
 transiently (image ~5 GB plus the ~3 GB uv download cache). Beszel alerts at 85%.
@@ -69,7 +70,13 @@ From v0.1.2 the packages install from a layer keyed on `uv export
 --no-emit-project`, so a bump that changes no dependency reuses that layer. Keep
 the layer cache (`docker builder prune --filter type=exec.cachemount` drops only
 the download cache); a full `docker builder prune` forces the next build to
-re-install everything.
+re-install everything. ⚠ **The cache did not survive on esh-ml1 for v0.1.3** (the
+deps layer re-ran for 1m43s, and the rootfs touched 90% until the old image was
+removed). On fv-ml1 the same build was CACHED (49 s), and the exported requirements
+were byte-identical across the two versions, so the layering itself works. The
+esh-ml1 miss is unexplained; suspect the cache pruning done there after v0.1.2.
+**On esh-ml1, check `df` before a build, and have ≥15 GB free or remove the old
+image first.**
 
 ## Startup is fail-closed on CUDA