vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(augaman): v0.1.3 on esh-ml1 and fv-ml1; after-bench: GPU ~3x faster, CPU mode regressed

Browse Source 
Both hosts are rebuilt from tag v0.1.3 (one ONNX session per detector canvas) and
redeployed. pytest -m gpu tests/vision passes 3/3 on each card.

Server-side for one face, same harness as the v0.1.2 baseline:
- esh-ml1 GPU 144 -> 48 ms
- fv-ml1 GPU 75 -> 27 ms
End to end from nh3-dev: 101.6 and 73.8 ms.

CPU mode got slower on every CPU target: fv-ml1 cpuset 0-5 went 152 -> 205 ms
with a face, and the no-face frame roughly doubled. That is well outside the
run-to-run spread. The suspected cause (not measured) is per-session ORT
thread pools spinning. Reported to augaman-dev. Neither deployment uses CPU
mode.

On esh-ml1 the dependency layer missed the build cache and the rootfs touched
90% until the v0.1.2 image was removed. fv-ml1's build hit the cache, and the
exported requirements are identical, so the stack README now says to check
disk before building on esh-ml1.
This commit is contained in:
vh
2026-09-27 01:13:01 -07:00
parent 317868dc7e
commit ef64a69e30
4 changed files with 589 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/pfi/augaman-speed-bench/README.md
+37 -3
View File
@@ -1,8 +1,42 @@
# augaman speed bench: CPU vs esh-ml1 GPU vs fv-ml1 GPU
Asked by Prime on 2026-09-27. This is the **v0.1.2 "before"** baseline. v0.1.3 gives
each detector canvas its own ONNX session, removing the ~90 ms CUDA shape-switching
cost that comfy-dev found. The same harness is to be re-run on v0.1.3 for the "after".
Asked by Prime on 2026-09-27. Two passes of the same harness:
**v0.1.2 (before, 0021–0026 PT)** and **v0.1.3 (after, 0106–0112 PT)**. v0.1.3 gives
each detector canvas its own ONNX session, which removes the ~90 ms CUDA
shape-switching cost that comfy-dev found.
## Headline: server-side ms per frame, one face (a) / no face (c)
| target | v0.1.2 | **v0.1.3** | change |
|---|---|---|---|
| esh-ml1 GPU (RTX 2000E) | 143.8 / 136.2 | **48.0 / 37.2** | ~3.0× faster |
| fv-ml1 GPU1 (RTX PRO 6000) | 75.0 / 69.5 | **27.3 / 22.9** | ~2.8× faster |
| fv-ml1 CPU, cpuset 0-5 | 152.4 / 72.3 | **205.1 / 133.2** | ⚠ ~1.35× / 1.8× SLOWER |
| fv-ml1 CPU, cpuset 0-23 | 155.9 / 61.9 | **204.6 / 124.0** | ⚠ SLOWER |
| esh-ml1 CPU, 6 LXC threads | 887.6 / 159.4 | **1105.1 / 395.4** | ⚠ SLOWER |
End to end from nh3-dev, frame (a) p50: esh-ml1 GPU **101.6 ms** (was 196.5) and fv-ml1
GPU **73.8 ms** (was 126.6). The /health floor is ~27 ms for both.
**The GPU gain is real, and CPU mode regressed.** The CPU slowdown sits far outside the
run-to-run spread: fv CPU6 (c) ran 158–166 ms against 111–113 before. It shows on
every CPU target and hits the no-face frame hardest, which points at the detector
sessions. Hypothesis, NOT measured: each of the three ORT sessions now has its own
intra-op thread pool, whose threads spin while another session runs, so the CPU
oversubscribes. The candidate fixes are `session.intra_op.allow_spinning=0` or a shared
global thread pool. Reported to augaman-dev. Neither deployment runs in CPU mode, so
nothing live is affected.
VRAM after the bench (nvidia-smi): esh-ml1 736 MiB (514 on v0.1.2), fv-ml1 1828 MiB
(1264 on v0.1.2). `pytest -m gpu tests/vision` passes 3/3 on both cards on v0.1.3
(three sessions proven: detector@128, detector@640, embedder). Raw data:
`rows-2026-09-27-v0.1.3.json`, and GPU utilisation samples every ~15 s in
`gpu-util-2026-09-27-v0.1.3.log` (mostly 0%, with peaks of 21% on esh-ml1 and 31%
on fv-ml1 GPU 1; fv-ml1 GPU 2 hit 99% once from another seat's traffic).
---
# v0.1.2 baseline detail
## Harness (it is part of the number)
docs/pfi/augaman-speed-bench/rows-2026-09-27-v0.1.3.json
+542
View File
@@ -0,0 +1,542 @@
[
{
"target": "esh-ml1 GPU",
"frame": "health_floor",
"run": 1,
"p50": 28.56831700046314,
"p90": 35.426961704797584,
"server_probe_mean": null,
"bad": 0
},
{
"target": "esh-ml1 GPU",
"frame": "a_1080p_1face",
"run": 1,
"p50": 103.24051550560398,
"p90": 108.98464530328056,
"server_probe_mean": 50.50277935079066,
"bad": 0
},
{
"target": "esh-ml1 GPU",
"frame": "b_960x1001_2face",
"run": 1,
"p50": 91.28140949906083,
"p90": 97.65329169604229,
"server_probe_mean": 36.06587980029872,
"bad": 0
},
{
"target": "esh-ml1 GPU",
"frame": "c_1080p_0face",
"run": 1,
"p50": 80.58081250055693,
"p90": 88.59687689837301,
"server_probe_mean": 37.25972330139484,
"bad": 0
},
{
"target": "fv-ml1 GPU1",
"frame": "health_floor",
"run": 1,
"p50": 31.64928349724505,
"p90": 33.86286070744973,
"server_probe_mean": null,
"bad": 0
},
{
"target": "fv-ml1 GPU1",
"frame": "a_1080p_1face",
"run": 1,
"p50": 77.40206849848619,
"p90": 87.3496351967333,
"server_probe_mean": 26.340403244830668,
"bad": 0
},
{
"target": "fv-ml1 GPU1",
"frame": "b_960x1001_2face",
"run": 1,
"p50": 73.34154599084286,
"p90": 77.04311340639833,
"server_probe_mean": 24.761526938527822,
"bad": 0
},
{
"target": "fv-ml1 GPU1",
"frame": "c_1080p_0face",
"run": 1,
"p50": 60.86771749687614,
"p90": 64.20872810704168,
"server_probe_mean": 22.897095198277384,
"bad": 0
},
{
"target": "esh-ml1 CPU6",
"frame": "health_floor",
"run": 1,
"p50": 31.80485299526481,
"p90": 37.67810769495554,
"server_probe_mean": null,
"bad": 0
},
{
"target": "esh-ml1 CPU6",
"frame": "a_1080p_1face",
"run": 1,
"p50": 1157.2700699980487,
"p90": 1285.2505169998042,
"server_probe_mean": 1105.1321487975656,
"bad": 0
},
{
"target": "esh-ml1 CPU6",
"frame": "b_960x1001_2face",
"run": 1,
"p50": 1620.7859505011584,
"p90": 1897.2889195065363,
"server_probe_mean": 1567.7373465063283,
"bad": 0
},
{
"target": "esh-ml1 CPU6",
"frame": "c_1080p_0face",
"run": 1,
"p50": 358.5786059993552,
"p90": 447.7066722058226,
"server_probe_mean": 325.5370168029913,
"bad": 0
},
{
"target": "fv-ml1 CPU6",
"frame": "health_floor",
"run": 1,
"p50": 29.225862497696653,
"p90": 31.134746607858688,
"server_probe_mean": null,
"bad": 0
},
{
"target": "fv-ml1 CPU6",
"frame": "a_1080p_1face",
"run": 1,
"p50": 244.82941399764968,
"p90": 285.62337980692973,
"server_probe_mean": 205.05198980681598,
"bad": 0
},
{
"target": "fv-ml1 CPU6",
"frame": "b_960x1001_2face",
"run": 1,
"p50": 273.4833980066469,
"p90": 318.48532019648707,
"server_probe_mean": 225.52827950567007,
"bad": 0
},
{
"target": "fv-ml1 CPU6",
"frame": "c_1080p_0face",
"run": 1,
"p50": 165.87754249485442,
"p90": 204.7811397977057,
"server_probe_mean": 133.18235542392358,
"bad": 0
},
{
"target": "fv-ml1 CPU24",
"frame": "health_floor",
"run": 1,
"p50": 28.081258496968076,
"p90": 32.582762498350355,
"server_probe_mean": null,
"bad": 0
},
{
"target": "fv-ml1 CPU24",
"frame": "a_1080p_1face",
"run": 1,
"p50": 247.0757969931583,
"p90": 265.46713550778804,
"server_probe_mean": 200.1803957624361,
"bad": 0
},
{
"target": "fv-ml1 CPU24",
"frame": "b_960x1001_2face",
"run": 1,
"p50": 273.32474349532276,
"p90": 343.1363677998889,
"server_probe_mean": 240.7834232551977,
"bad": 0
},
{
"target": "fv-ml1 CPU24",
"frame": "c_1080p_0face",
"run": 1,
"p50": 166.3104715044028,
"p90": 182.07621549954638,
"server_probe_mean": 124.0462520509027,
"bad": 0
},
{
"target": "esh-ml1 GPU",
"frame": "health_floor",
"run": 2,
"p50": 26.54249199986225,
"p90": 27.6983294999809,
"server_probe_mean": null,
"bad": 0
},
{
"target": "esh-ml1 GPU",
"frame": "a_1080p_1face",
"run": 2,
"p50": 101.5832139964914,
"p90": 109.36262830364285,
"server_probe_mean": 47.99011784780305,
"bad": 0
},
{
"target": "esh-ml1 GPU",
"frame": "b_960x1001_2face",
"run": 2,
"p50": 94.75307750108186,
"p90": 100.28928269312019,
"server_probe_mean": 36.026441550347954,
"bad": 0
},
{
"target": "esh-ml1 GPU",
"frame": "c_1080p_0face",
"run": 2,
"p50": 78.17531850014348,
"p90": 83.03192700113868,
"server_probe_mean": 37.16747584985569,
"bad": 0
},
{
"target": "fv-ml1 GPU1",
"frame": "health_floor",
"run": 2,
"p50": 26.91060849610949,
"p90": 27.50819110951852,
"server_probe_mean": null,
"bad": 0
},
{
"target": "fv-ml1 GPU1",
"frame": "a_1080p_1face",
"run": 2,
"p50": 73.84634450136218,
"p90": 75.3575598981115,
"server_probe_mean": 27.809194929432124,
"bad": 0
},
{
"target": "fv-ml1 GPU1",
"frame": "b_960x1001_2face",
"run": 2,
"p50": 73.40573299734388,
"p90": 76.6928782017203,
"server_probe_mean": 25.148114131297916,
"bad": 0
},
{
"target": "fv-ml1 GPU1",
"frame": "c_1080p_0face",
"run": 2,
"p50": 60.866275503940415,
"p90": 64.77840160223423,
"server_probe_mean": 21.837973268702626,
"bad": 0
},
{
"target": "esh-ml1 CPU6",
"frame": "health_floor",
"run": 2,
"p50": 30.955315494793467,
"p90": 34.70198850845919,
"server_probe_mean": null,
"bad": 0
},
{
"target": "esh-ml1 CPU6",
"frame": "a_1080p_1face",
"run": 2,
"p50": 1048.8297365009203,
"p90": 1299.0026480983945,
"server_probe_mean": 1022.0766173981247,
"bad": 0
},
{
"target": "esh-ml1 CPU6",
"frame": "b_960x1001_2face",
"run": 2,
"p50": 1584.0665085052024,
"p90": 1715.8558561946847,
"server_probe_mean": 1513.3129546549753,
"bad": 0
},
{
"target": "esh-ml1 CPU6",
"frame": "c_1080p_0face",
"run": 2,
"p50": 433.8740794992191,
"p90": 538.4446465948713,
"server_probe_mean": 395.367383849225,
"bad": 0
},
{
"target": "fv-ml1 CPU6",
"frame": "health_floor",
"run": 2,
"p50": 29.1177155013429,
"p90": 31.4779651016579,
"server_probe_mean": null,
"bad": 0
},
{
"target": "fv-ml1 CPU6",
"frame": "a_1080p_1face",
"run": 2,
"p50": 258.8035505032167,
"p90": 279.1849860892398,
"server_probe_mean": 204.92596072144806,
"bad": 0
},
{
"target": "fv-ml1 CPU6",
"frame": "b_960x1001_2face",
"run": 2,
"p50": 253.7849774962524,
"p90": 302.2328945007757,
"server_probe_mean": 216.03364354232326,
"bad": 0
},
{
"target": "fv-ml1 CPU6",
"frame": "c_1080p_0face",
"run": 2,
"p50": 159.76047250296688,
"p90": 347.6618904911466,
"server_probe_mean": 378.62427223008126,
"bad": 0
},
{
"target": "fv-ml1 CPU24",
"frame": "health_floor",
"run": 2,
"p50": 28.286710505199153,
"p90": 33.18228400457884,
"server_probe_mean": null,
"bad": 0
},
{
"target": "fv-ml1 CPU24",
"frame": "a_1080p_1face",
"run": 2,
"p50": 250.35845449747285,
"p90": 270.55101390578784,
"server_probe_mean": 204.57896552979946,
"bad": 0
},
{
"target": "fv-ml1 CPU24",
"frame": "b_960x1001_2face",
"run": 2,
"p50": 276.2931550023495,
"p90": 302.7462427984574,
"server_probe_mean": 228.16327152540907,
"bad": 0
},
{
"target": "fv-ml1 CPU24",
"frame": "c_1080p_0face",
"run": 2,
"p50": 173.82631749933353,
"p90": 203.72216301038867,
"server_probe_mean": 144.8986719478853,
"bad": 0
},
{
"target": "esh-ml1 GPU",
"frame": "health_floor",
"run": 3,
"p50": 26.987262994225603,
"p90": 30.949878702813294,
"server_probe_mean": null,
"bad": 0
},
{
"target": "esh-ml1 GPU",
"frame": "a_1080p_1face",
"run": 3,
"p50": 99.26728250138694,
"p90": 105.58418770378921,
"server_probe_mean": 45.727238450490404,
"bad": 0
},
{
"target": "esh-ml1 GPU",
"frame": "b_960x1001_2face",
"run": 3,
"p50": 92.11020849761553,
"p90": 96.73994110344212,
"server_probe_mean": 34.403727897733916,
"bad": 0
},
{
"target": "esh-ml1 GPU",
"frame": "c_1080p_0face",
"run": 3,
"p50": 79.78740399994422,
"p90": 85.02629639551742,
"server_probe_mean": 37.2430077943136,
"bad": 0
},
{
"target": "fv-ml1 GPU1",
"frame": "health_floor",
"run": 3,
"p50": 27.467196501675062,
"p90": 32.5130715049454,
"server_probe_mean": null,
"bad": 0
},
{
"target": "fv-ml1 GPU1",
"frame": "a_1080p_1face",
"run": 3,
"p50": 73.05359399470035,
"p90": 79.36581129324622,
"server_probe_mean": 27.340552653186023,
"bad": 0
},
{
"target": "fv-ml1 GPU1",
"frame": "b_960x1001_2face",
"run": 3,
"p50": 72.0015334954951,
"p90": 75.61690630245721,
"server_probe_mean": 24.976778693962842,
"bad": 0
},
{
"target": "fv-ml1 GPU1",
"frame": "c_1080p_0face",
"run": 3,
"p50": 61.28011599503225,
"p90": 63.8927058011177,
"server_probe_mean": 22.860179003328085,
"bad": 0
},
{
"target": "esh-ml1 CPU6",
"frame": "health_floor",
"run": 3,
"p50": 28.69356150040403,
"p90": 30.96141718851868,
"server_probe_mean": null,
"bad": 0
},
{
"target": "esh-ml1 CPU6",
"frame": "a_1080p_1face",
"run": 3,
"p50": 1115.432997001335,
"p90": 1451.1805613117758,
"server_probe_mean": 1106.370482649072,
"bad": 0
},
{
"target": "esh-ml1 CPU6",
"frame": "b_960x1001_2face",
"run": 3,
"p50": 1618.1769984978018,
"p90": 1854.3222563093877,
"server_probe_mean": 1573.5961115453392,
"bad": 0
},
{
"target": "esh-ml1 CPU6",
"frame": "c_1080p_0face",
"run": 3,
"p50": 449.7984624977107,
"p90": 541.9355103018461,
"server_probe_mean": 404.3181982007809,
"bad": 0
},
{
"target": "fv-ml1 CPU6",
"frame": "health_floor",
"run": 3,
"p50": 29.24010700371582,
"p90": 33.91167670779396,
"server_probe_mean": null,
"bad": 0
},
{
"target": "fv-ml1 CPU6",
"frame": "a_1080p_1face",
"run": 3,
"p50": 248.83804350247374,
"p90": 305.8311790940934,
"server_probe_mean": 207.7646869351156,
"bad": 0
},
{
"target": "fv-ml1 CPU6",
"frame": "b_960x1001_2face",
"run": 3,
"p50": 252.01708450185833,
"p90": 323.74196870077867,
"server_probe_mean": 215.00399404903874,
"bad": 0
},
{
"target": "fv-ml1 CPU6",
"frame": "c_1080p_0face",
"run": 3,
"p50": 158.14820949890418,
"p90": 203.81181769480463,
"server_probe_mean": 131.31015392718837,
"bad": 0
},
{
"target": "fv-ml1 CPU24",
"frame": "health_floor",
"run": 3,
"p50": 29.235157504444942,
"p90": 33.62159581010929,
"server_probe_mean": null,
"bad": 0
},
{
"target": "fv-ml1 CPU24",
"frame": "a_1080p_1face",
"run": 3,
"p50": 251.2742774997605,
"p90": 272.51768180867657,
"server_probe_mean": 206.39763702638447,
"bad": 0
},
{
"target": "fv-ml1 CPU24",
"frame": "b_960x1001_2face",
"run": 3,
"p50": 265.6091139942873,
"p90": 297.4889616059954,
"server_probe_mean": 222.6791250752285,
"bad": 0
},
{
"target": "fv-ml1 CPU24",
"frame": "c_1080p_0face",
"run": 3,
"p50": 158.92017900478095,
"p90": 181.67336140904808,
"server_probe_mean": 121.74809173448011,
"bad": 0
}
]