vh
ef64a69e30
Both hosts are rebuilt from tag v0.1.3 (one ONNX session per detector canvas) and redeployed. pytest -m gpu tests/vision passes 3/3 on each card. Server-side for one face, same harness as the v0.1.2 baseline: - esh-ml1 GPU 144 -> 48 ms - fv-ml1 GPU 75 -> 27 ms End to end from nh3-dev: 101.6 and 73.8 ms. CPU mode got slower on every CPU target: fv-ml1 cpuset 0-5 went 152 -> 205 ms with a face, and the no-face frame roughly doubled. That is well outside the run-to-run spread. The suspected cause (not measured) is per-session ORT thread pools spinning. Reported to augaman-dev. Neither deployment uses CPU mode. On esh-ml1 the dependency layer missed the build cache and the rootfs touched 90% until the v0.1.2 image was removed. fv-ml1's build hit the cache, and the exported requirements are identical, so the stack README now says to check disk before building on esh-ml1.
114 lines
6.8 KiB
Markdown
114 lines
6.8 KiB
Markdown
# augaman speed bench: CPU vs esh-ml1 GPU vs fv-ml1 GPU
|
||
|
||
Asked by Prime on 2026-09-27. Two passes of the same harness:
|
||
**v0.1.2 (before, 0021–0026 PT)** and **v0.1.3 (after, 0106–0112 PT)**. v0.1.3 gives
|
||
each detector canvas its own ONNX session, which removes the ~90 ms CUDA
|
||
shape-switching cost that comfy-dev found.
|
||
|
||
## Headline: server-side ms per frame, one face (a) / no face (c)
|
||
|
||
| target | v0.1.2 | **v0.1.3** | change |
|
||
|---|---|---|---|
|
||
| esh-ml1 GPU (RTX 2000E) | 143.8 / 136.2 | **48.0 / 37.2** | ~3.0× faster |
|
||
| fv-ml1 GPU1 (RTX PRO 6000) | 75.0 / 69.5 | **27.3 / 22.9** | ~2.8× faster |
|
||
| fv-ml1 CPU, cpuset 0-5 | 152.4 / 72.3 | **205.1 / 133.2** | ⚠ ~1.35× / 1.8× SLOWER |
|
||
| fv-ml1 CPU, cpuset 0-23 | 155.9 / 61.9 | **204.6 / 124.0** | ⚠ SLOWER |
|
||
| esh-ml1 CPU, 6 LXC threads | 887.6 / 159.4 | **1105.1 / 395.4** | ⚠ SLOWER |
|
||
|
||
End to end from nh3-dev, frame (a) p50: esh-ml1 GPU **101.6 ms** (was 196.5) and fv-ml1
|
||
GPU **73.8 ms** (was 126.6). The /health floor is ~27 ms for both.
|
||
|
||
**The GPU gain is real, and CPU mode regressed.** The CPU slowdown sits far outside the
|
||
run-to-run spread: fv CPU6 (c) ran 158–166 ms against 111–113 before. It shows on
|
||
every CPU target and hits the no-face frame hardest, which points at the detector
|
||
sessions. Hypothesis, NOT measured: each of the three ORT sessions now has its own
|
||
intra-op thread pool, whose threads spin while another session runs, so the CPU
|
||
oversubscribes. The candidate fixes are `session.intra_op.allow_spinning=0` or a shared
|
||
global thread pool. Reported to augaman-dev. Neither deployment runs in CPU mode, so
|
||
nothing live is affected.
|
||
|
||
VRAM after the bench (nvidia-smi): esh-ml1 736 MiB (514 on v0.1.2), fv-ml1 1828 MiB
|
||
(1264 on v0.1.2). `pytest -m gpu tests/vision` passes 3/3 on both cards on v0.1.3
|
||
(three sessions proven: detector@128, detector@640, embedder). Raw data:
|
||
`rows-2026-09-27-v0.1.3.json`, and GPU utilisation samples every ~15 s in
|
||
`gpu-util-2026-09-27-v0.1.3.log` (mostly 0%, with peaks of 21% on esh-ml1 and 31%
|
||
on fv-ml1 GPU 1; fv-ml1 GPU 2 hit 99% once from another seat's traffic).
|
||
|
||
---
|
||
|
||
# v0.1.2 baseline detail
|
||
|
||
## Harness (it is part of the number)
|
||
|
||
- **Client:** `bench.py` on nh3-dev, sequential. Each request uses a **fresh
|
||
connection** (no keep-alive) and is timed around the whole POST.
|
||
- **Protocol:** 3 runs. Per (target, frame, run) there are 3 warm-up requests,
|
||
then 20 timed ones. Runs are interleaved across targets and frames, so drift
|
||
spreads over every condition. Time: 0021–0026 PT, 2026-09-27.
|
||
- **Server-side column:** the change in `augaman_pipeline_seconds{op="probe"}`
|
||
sum/count over the 20 timed requests. It is decode + detect + embed with no
|
||
network or HTTP, and it is **the device comparison**. It is a mean, not a p50.
|
||
- **Floor:** a `/health` round trip, same client. It came out 28–34 ms, higher than
|
||
augaman-dev's 16 ms because of the fresh connection per request.
|
||
- **Frames** (`make_frames.py`): (a) 1920x1080 grey canvas with one face;
|
||
(b) `obama_biden_2015.jpg` as-is, 960x1001, two faces; **(c) the same 1080p grey
|
||
canvas with no face**, a detector-only control. Every response was checked for
|
||
the expected face count, and **0 of 900 were wrong**.
|
||
- **Images:** `augaman:0.1.2`, built from the same tag on each host (separate
|
||
builds, same source).
|
||
- **Targets:**
|
||
|
||
| target | device | notes |
|
||
|---|---|---|
|
||
| esh-ml1 GPU | RTX 2000E Ada 16 GB (50 W) | the production instance; shares the card with TEI + vllm-reward |
|
||
| fv-ml1 GPU1 | RTX PRO 6000 Blackwell Max-Q | the new second instance; shares GPU 1 with coder, erp and meromero seats |
|
||
| esh-ml1 CPU6 | the LXC's 6 host threads | throwaway `AUGAMAN_DEVICE=cpu` container. pve moved the LXC's cpuset between two reads, so it ran unpinned, with an affinity of 6 |
|
||
| fv-ml1 CPU6 | EPYC 9254, cpuset 0-5 | throwaway container; physical cores on socket 0 |
|
||
| fv-ml1 CPU24 | EPYC 9254, cpuset 0-23 | throwaway container; all physical cores of socket 0 |
|
||
|
||
GPU utilisation read 0% on fv-ml1 GPU 1 and 0–9% on esh-ml1 across the samples
|
||
(0020–0021, 3 samples each), so the co-resident seats were idle.
|
||
|
||
## Results (ms): median of the 3 run p50s, [min–max of the run p50s]
|
||
|
||
| target | (a) 1 face, e2e | (b) 2 faces, e2e | (c) no face, e2e | **server (a)** | **server (b)** | **server (c)** |
|
||
|---|---|---|---|---|---|---|
|
||
| esh-ml1 GPU | 196.5 [195.0–196.6] | 195.1 [194.0–196.1] | 181.0 [177.6–182.9] | **143.8** | **138.7** | **136.2** |
|
||
| fv-ml1 GPU1 | 126.6 [121.8–128.3] | 122.7 [122.4–127.9] | 109.2 [108.7–111.0] | **75.0** | **73.9** | **69.5** |
|
||
| esh-ml1 CPU6 | 954.6 [792.8–1023.3] | 1325.6 [1306.9–1327.6] | 205.1 [202.1–228.0] | **887.6** | **1253.4** | **159.4** |
|
||
| fv-ml1 CPU6 | 203.6 [203.5–204.5] | 222.5 [216.0–236.4] | 111.6 [110.6–112.7] | **152.4** | **179.4** | **72.3** |
|
||
| fv-ml1 CPU24 | 208.1 [204.6–221.6] | 223.7 [219.8–237.1] | 101.1 [100.5–113.4] | **155.9** | **167.3** | **61.9** |
|
||
|
||
The /health floor was 27.6–33.7 ms (median of run p50s) on every target. The raw
|
||
rows are in `rows-2026-09-27.json`, and `summarize.py` reproduces this table.
|
||
|
||
## What it says
|
||
|
||
- **fv-ml1's Blackwell is ~2× esh-ml1's Ada on v0.1.2:** 75 vs 144 ms server-side
|
||
for one face. End to end from nh3-dev it is 127 vs 197 ms.
|
||
- **On v0.1.2, a GPU buys little over a good CPU.** 6 EPYC cores do one face in
|
||
152 ms, as fast as esh-ml1's GPU (144 ms). With no face, fv-ml1's CPU (62–72 ms)
|
||
matches fv-ml1's GPU (70 ms). The GPU pipeline is dominated by something other
|
||
than GPU compute, and the cards read 0–9% utilisation while it ran. That is
|
||
consistent with the detector's shape-switching cost that v0.1.3 removes.
|
||
- **esh-ml1's CPU is not a viable home:** 0.9 s for one face and 1.3 s for two. Its
|
||
embedder costs ~730 ms per face, against ~80 ms on 6 EPYC cores. That gap is much
|
||
larger than the detector gap (159 vs 72 ms), and its runs were the noisiest
|
||
(793–1023 ms), so treat the esh CPU number as indicative. It was also sharing its 6
|
||
threads with TEI.
|
||
- **More EPYC threads do not help one frame:** CPU24 is roughly equal to CPU6 on faces.
|
||
|
||
## Controls and sensitivity
|
||
|
||
- **Repeats:** 3 runs × 20 per cell. The A-vs-A spread is the min–max above.
|
||
- **Noise floor:** the GPU cells spread by ≤6 ms across runs, and the fv CPU cells
|
||
by ≤20 ms. The esh-ml1 CPU (a) cell spread by 230 ms. Differences smaller than
|
||
those spreads are not findings.
|
||
- **Positive control (instrument agreement):** augaman-dev measured esh-ml1 (a)
|
||
independently at 186 ms with a 16 ms floor. This harness gets 196.5 ms with a
|
||
28 ms floor. Minus each floor, 170 vs 169 ms: they agree.
|
||
- **Positive control (detects work):** frame (c) is fastest on every target, and
|
||
the gap from (a) to (c) is the embedding cost. It is visible on every device.
|
||
- **Negative control:** the face count was checked on all 900 responses, and none
|
||
were wrong.
|