vh
ef64a69e30
Both hosts are rebuilt from tag v0.1.3 (one ONNX session per detector canvas) and redeployed. pytest -m gpu tests/vision passes 3/3 on each card. Server-side for one face, same harness as the v0.1.2 baseline: - esh-ml1 GPU 144 -> 48 ms - fv-ml1 GPU 75 -> 27 ms End to end from nh3-dev: 101.6 and 73.8 ms. CPU mode got slower on every CPU target: fv-ml1 cpuset 0-5 went 152 -> 205 ms with a face, and the no-face frame roughly doubled. That is well outside the run-to-run spread. The suspected cause (not measured) is per-session ORT thread pools spinning. Reported to augaman-dev. Neither deployment uses CPU mode. On esh-ml1 the dependency layer missed the build cache and the rootfs touched 90% until the v0.1.2 image was removed. fv-ml1's build hit the cache, and the exported requirements are identical, so the stack README now says to check disk before building on esh-ml1.
6.8 KiB
augaman speed bench: CPU vs esh-ml1 GPU vs fv-ml1 GPU
Asked by Prime on 2026-09-27. Two passes of the same harness: v0.1.2 (before, 0021–0026 PT) and v0.1.3 (after, 0106–0112 PT). v0.1.3 gives each detector canvas its own ONNX session, which removes the ~90 ms CUDA shape-switching cost that comfy-dev found.
Headline: server-side ms per frame, one face (a) / no face (c)
| target | v0.1.2 | v0.1.3 | change |
|---|---|---|---|
| esh-ml1 GPU (RTX 2000E) | 143.8 / 136.2 | 48.0 / 37.2 | ~3.0× faster |
| fv-ml1 GPU1 (RTX PRO 6000) | 75.0 / 69.5 | 27.3 / 22.9 | ~2.8× faster |
| fv-ml1 CPU, cpuset 0-5 | 152.4 / 72.3 | 205.1 / 133.2 | ⚠ ~1.35× / 1.8× SLOWER |
| fv-ml1 CPU, cpuset 0-23 | 155.9 / 61.9 | 204.6 / 124.0 | ⚠ SLOWER |
| esh-ml1 CPU, 6 LXC threads | 887.6 / 159.4 | 1105.1 / 395.4 | ⚠ SLOWER |
End to end from nh3-dev, frame (a) p50: esh-ml1 GPU 101.6 ms (was 196.5) and fv-ml1 GPU 73.8 ms (was 126.6). The /health floor is ~27 ms for both.
The GPU gain is real, and CPU mode regressed. The CPU slowdown sits far outside the
run-to-run spread: fv CPU6 (c) ran 158–166 ms against 111–113 before. It shows on
every CPU target and hits the no-face frame hardest, which points at the detector
sessions. Hypothesis, NOT measured: each of the three ORT sessions now has its own
intra-op thread pool, whose threads spin while another session runs, so the CPU
oversubscribes. The candidate fixes are session.intra_op.allow_spinning=0 or a shared
global thread pool. Reported to augaman-dev. Neither deployment runs in CPU mode, so
nothing live is affected.
VRAM after the bench (nvidia-smi): esh-ml1 736 MiB (514 on v0.1.2), fv-ml1 1828 MiB
(1264 on v0.1.2). pytest -m gpu tests/vision passes 3/3 on both cards on v0.1.3
(three sessions proven: detector@128, detector@640, embedder). Raw data:
rows-2026-09-27-v0.1.3.json, and GPU utilisation samples every ~15 s in
gpu-util-2026-09-27-v0.1.3.log (mostly 0%, with peaks of 21% on esh-ml1 and 31%
on fv-ml1 GPU 1; fv-ml1 GPU 2 hit 99% once from another seat's traffic).
v0.1.2 baseline detail
Harness (it is part of the number)
- Client:
bench.pyon nh3-dev, sequential. Each request uses a fresh connection (no keep-alive) and is timed around the whole POST. - Protocol: 3 runs. Per (target, frame, run) there are 3 warm-up requests, then 20 timed ones. Runs are interleaved across targets and frames, so drift spreads over every condition. Time: 0021–0026 PT, 2026-09-27.
- Server-side column: the change in
augaman_pipeline_seconds{op="probe"}sum/count over the 20 timed requests. It is decode + detect + embed with no network or HTTP, and it is the device comparison. It is a mean, not a p50. - Floor: a
/healthround trip, same client. It came out 28–34 ms, higher than augaman-dev's 16 ms because of the fresh connection per request. - Frames (
make_frames.py): (a) 1920x1080 grey canvas with one face; (b)obama_biden_2015.jpgas-is, 960x1001, two faces; (c) the same 1080p grey canvas with no face, a detector-only control. Every response was checked for the expected face count, and 0 of 900 were wrong. - Images:
augaman:0.1.2, built from the same tag on each host (separate builds, same source). - Targets:
| target | device | notes |
|---|---|---|
| esh-ml1 GPU | RTX 2000E Ada 16 GB (50 W) | the production instance; shares the card with TEI + vllm-reward |
| fv-ml1 GPU1 | RTX PRO 6000 Blackwell Max-Q | the new second instance; shares GPU 1 with coder, erp and meromero seats |
| esh-ml1 CPU6 | the LXC's 6 host threads | throwaway AUGAMAN_DEVICE=cpu container. pve moved the LXC's cpuset between two reads, so it ran unpinned, with an affinity of 6 |
| fv-ml1 CPU6 | EPYC 9254, cpuset 0-5 | throwaway container; physical cores on socket 0 |
| fv-ml1 CPU24 | EPYC 9254, cpuset 0-23 | throwaway container; all physical cores of socket 0 |
GPU utilisation read 0% on fv-ml1 GPU 1 and 0–9% on esh-ml1 across the samples (0020–0021, 3 samples each), so the co-resident seats were idle.
Results (ms): median of the 3 run p50s, [min–max of the run p50s]
| target | (a) 1 face, e2e | (b) 2 faces, e2e | (c) no face, e2e | server (a) | server (b) | server (c) |
|---|---|---|---|---|---|---|
| esh-ml1 GPU | 196.5 [195.0–196.6] | 195.1 [194.0–196.1] | 181.0 [177.6–182.9] | 143.8 | 138.7 | 136.2 |
| fv-ml1 GPU1 | 126.6 [121.8–128.3] | 122.7 [122.4–127.9] | 109.2 [108.7–111.0] | 75.0 | 73.9 | 69.5 |
| esh-ml1 CPU6 | 954.6 [792.8–1023.3] | 1325.6 [1306.9–1327.6] | 205.1 [202.1–228.0] | 887.6 | 1253.4 | 159.4 |
| fv-ml1 CPU6 | 203.6 [203.5–204.5] | 222.5 [216.0–236.4] | 111.6 [110.6–112.7] | 152.4 | 179.4 | 72.3 |
| fv-ml1 CPU24 | 208.1 [204.6–221.6] | 223.7 [219.8–237.1] | 101.1 [100.5–113.4] | 155.9 | 167.3 | 61.9 |
The /health floor was 27.6–33.7 ms (median of run p50s) on every target. The raw
rows are in rows-2026-09-27.json, and summarize.py reproduces this table.
What it says
- fv-ml1's Blackwell is ~2× esh-ml1's Ada on v0.1.2: 75 vs 144 ms server-side for one face. End to end from nh3-dev it is 127 vs 197 ms.
- On v0.1.2, a GPU buys little over a good CPU. 6 EPYC cores do one face in 152 ms, as fast as esh-ml1's GPU (144 ms). With no face, fv-ml1's CPU (62–72 ms) matches fv-ml1's GPU (70 ms). The GPU pipeline is dominated by something other than GPU compute, and the cards read 0–9% utilisation while it ran. That is consistent with the detector's shape-switching cost that v0.1.3 removes.
- esh-ml1's CPU is not a viable home: 0.9 s for one face and 1.3 s for two. Its embedder costs ~730 ms per face, against ~80 ms on 6 EPYC cores. That gap is much larger than the detector gap (159 vs 72 ms), and its runs were the noisiest (793–1023 ms), so treat the esh CPU number as indicative. It was also sharing its 6 threads with TEI.
- More EPYC threads do not help one frame: CPU24 is roughly equal to CPU6 on faces.
Controls and sensitivity
- Repeats: 3 runs × 20 per cell. The A-vs-A spread is the min–max above.
- Noise floor: the GPU cells spread by ≤6 ms across runs, and the fv CPU cells by ≤20 ms. The esh-ml1 CPU (a) cell spread by 230 ms. Differences smaller than those spreads are not findings.
- Positive control (instrument agreement): augaman-dev measured esh-ml1 (a) independently at 186 ms with a 16 ms floor. This harness gets 196.5 ms with a 28 ms floor. Minus each floor, 170 vs 169 ms: they agree.
- Positive control (detects work): frame (c) is fastest on every target, and the gap from (a) to (c) is the embedding cost. It is visible on every device.
- Negative control: the face count was checked on all 900 responses, and none were wrong.