From 3e73374c6e47d948d8af5bbe61a8a7ab0298bb25 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 25 Sep 2026 06:40:12 -0700 Subject: [PATCH] =?UTF-8?q?docs(esh-ml1):=20speed=20A/B=20vs=20fv-ml1=20?= =?UTF-8?q?=E2=80=94=20single=20queries=20a=20wash,=20bulk=203-10x=20slowe?= =?UTF-8?q?r;=20stay=20failover-only?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../2026-09-24-esh-ml1-embed-rerank.md | 7 ++++++ servers/esh-ml1/README.md | 23 +++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md b/persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md index 75381c7..26ade76 100644 --- a/persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md +++ b/persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md @@ -60,3 +60,10 @@ an FV-outage failover for Worldtree + nevermore, not local service for ESH. ESH consumers, which are idle — recommended parked. esh-ml1 not yet in Homepage docker.yaml or Beszel. The ~18.7 s host-down failover penalty is untuned (router retries/cooldown). + +**Speed A/B vs fv-ml1 (2026-09-25 0628–0640 PT, table in `servers/esh-ml1/README.md`).** +Single queries: a wash (on-box ESH ~9 ms vs FV ~12 ms; from ana-docker ESH is +faster because it is network-closer). Bulk: ESH 3–10× slower (bulk embed ~50 vs +~403 passages/s; rerank-20 ~170 vs ~51 ms p50, ~5.7 vs ~55 req/s at conc 8). +Noise floor 3–5% on throughput. ⇒ failover role confirmed; load-sharing would +roughly double rerank latency for half the calls. diff --git a/servers/esh-ml1/README.md b/servers/esh-ml1/README.md index 0a55fb6..515d77e 100644 --- a/servers/esh-ml1/README.md +++ b/servers/esh-ml1/README.md @@ -42,6 +42,29 @@ The cross-site difference is inside each site's own run-to-run noise; this method cannot resolve a cosine gap below ~2×10⁻⁴. Reranker scores differed by at most 0.000145 (FV-vs-FV floor 0.000181), with identical ranking. +**Speed vs fv-ml1, measured 2026-09-25 0628–0640 PT** (same vLLM v0.24.0 and +flags; fv-ml1 GPU 1 is shared with other seats but read 0% util before and +after; synthetic text; each cell = median of 6 reps = 2 interleaved runs × 3): + +| workload | on-box FV | on-box ESH | from ana-docker FV | from ana-docker ESH | +|---|---|---|---|---| +| embed 1 query, p50 | ~12 ms | ~9 ms | ~29 ms | ~23 ms | +| embed 1 × ~512 tok, p50 | ~15 ms | ~23 ms | ~31 ms | ~31 ms | +| embed 1 × ~2k tok, p50 | ~25 ms | ~86 ms | ~48 ms | ~90 ms | +| bulk embed, passages/s (64×512 tok, conc 4) | ~403 | ~50 | ~374 | ~51 | +| rerank 20 docs, p50 | ~51 ms | ~170 ms | ~82 ms | ~175 ms | +| rerank 20 docs, req/s at conc 8 | ~55 | ~5.7 | ~52 | ~5.9 | + +Read it as: **single queries are a wash** (ESH is ~6 ms faster from the gateway +because it is closer — 6 ms RTT vs 17.6 ms), **anything bulk is 3–10× slower** +on the 50 W Ada. Noise floor (run vs run, same site): ~3–5% on throughput, ±3 ms +on small-query latency on fv-ml1, so the small-query rows cannot rank the sites. +Positive control: 2k-token inputs were slower than 512-token ones on both +sites. One fv-ml1 rep of the gateway-vantage rerank burst stalled to 1.0 req/s +(64 requests in ~64 s); it did not recur in 11 other reps — unexplained, n=1. +**Consequence: keep esh-ml1 as failover, don't load-share** — sharing would +roughly double rerank latency for half the fleet's calls. + **VRAM:** 0.20 × 16,380 MiB each; 4,823 MiB in use with both loaded, ~11 GB free.