docs(esh-ml1): speed A/B vs fv-ml1 — single queries a wash, bulk 3-10x slower; stay failover-only

persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md

@@ -60,3 +60,10 @@ an FV-outage failover for Worldtree + nevermore, not local service for ESH.
 ESH consumers, which are idle — recommended parked. esh-ml1 not yet in
 Homepage docker.yaml or Beszel. The ~18.7 s host-down failover penalty is
 untuned (router retries/cooldown).
+
+**Speed A/B vs fv-ml1 (2026-09-25 0628–0640 PT, table in `servers/esh-ml1/README.md`).**
+Single queries: a wash (on-box ESH ~9 ms vs FV ~12 ms; from ana-docker ESH is
+faster because it is network-closer). Bulk: ESH 3–10× slower (bulk embed ~50 vs
+~403 passages/s; rerank-20 ~170 vs ~51 ms p50, ~5.7 vs ~55 req/s at conc 8).
+Noise floor 3–5% on throughput. ⇒ failover role confirmed; load-sharing would
+roughly double rerank latency for half the calls.

servers/esh-ml1/README.md

@@ -42,6 +42,29 @@ The cross-site difference is inside each site's own run-to-run noise; this
 method cannot resolve a cosine gap below ~2×10⁻⁴. Reranker scores differed by
 at most 0.000145 (FV-vs-FV floor 0.000181), with identical ranking.
 
+**Speed vs fv-ml1, measured 2026-09-25 0628–0640 PT** (same vLLM v0.24.0 and
+flags; fv-ml1 GPU 1 is shared with other seats but read 0% util before and
+after; synthetic text; each cell = median of 6 reps = 2 interleaved runs × 3):
+
+| workload | on-box FV | on-box ESH | from ana-docker FV | from ana-docker ESH |
+|---|---|---|---|---|
+| embed 1 query, p50 | ~12 ms | ~9 ms | ~29 ms | ~23 ms |
+| embed 1 × ~512 tok, p50 | ~15 ms | ~23 ms | ~31 ms | ~31 ms |
+| embed 1 × ~2k tok, p50 | ~25 ms | ~86 ms | ~48 ms | ~90 ms |
+| bulk embed, passages/s (64×512 tok, conc 4) | ~403 | ~50 | ~374 | ~51 |
+| rerank 20 docs, p50 | ~51 ms | ~170 ms | ~82 ms | ~175 ms |
+| rerank 20 docs, req/s at conc 8 | ~55 | ~5.7 | ~52 | ~5.9 |
+
+Read it as: **single queries are a wash** (ESH is ~6 ms faster from the gateway
+because it is closer — 6 ms RTT vs 17.6 ms), **anything bulk is 3–10× slower**
+on the 50 W Ada. Noise floor (run vs run, same site): ~3–5% on throughput, ±3 ms
+on small-query latency on fv-ml1, so the small-query rows cannot rank the sites.
+Positive control: 2k-token inputs were slower than 512-token ones on both
+sites. One fv-ml1 rep of the gateway-vantage rerank burst stalled to 1.0 req/s
+(64 requests in ~64 s); it did not recur in 11 other reps — unexplained, n=1.
+**Consequence: keep esh-ml1 as failover, don't load-share** — sharing would
+roughly double rerank latency for half the fleet's calls.
+
 **VRAM:** 0.20 × 16,380 MiB each; 4,823 MiB in use with both loaded, ~11 GB
 free.