diff --git a/servers/esh-ml1/README.md b/servers/esh-ml1/README.md index 515d77e..d7260a7 100644 --- a/servers/esh-ml1/README.md +++ b/servers/esh-ml1/README.md @@ -65,6 +65,22 @@ sites. One fv-ml1 rep of the gateway-vantage rerank burst stalled to 1.0 req/s **Consequence: keep esh-ml1 as failover, don't load-share** — sharing would roughly double rerank latency for half the fleet's calls. +**Whole-novel embedding, measured 2026-09-25 0653–0717 PT** — *The Stand* +(uncut, ~1,150 pages): 470,783 words, 12,814 `

` paragraphs (median 22 words), +617,833 tokens. Client on nh3-dev (where worldtree-gateway runs); 3 reps each: + +| how | fv-ml1 | esh-ml1 | +|---|---|---| +| via gateway, 1 paragraph per request (1,000-para sample, scaled) | 59.8 ms/para → ~12.8 min | 55.3 ms/para → ~11.8 min | +| via gateway, 64 per request, 1 at a time | 100–123 s | 91–100 s | +| via gateway, 64 per request, 4 in flight | 46–54 s | 45–50 s | +| direct to the seat, 64 per request, 4 in flight | 13.4–14.1 s | 28.8–29.6 s | + +Through the gateway the GPU is not the bottleneck: one-per-request is network + +LiteLLM overhead, and batched runs pinned LiteLLM at ~76% CPU (64×1024-float +JSON per response). Only the direct path shows the card: esh-ml1 is ~2.2× +slower, ~21k vs ~45k tokens/s. + **VRAM:** 0.20 × 16,380 MiB each; 4,823 MiB in use with both loaded, ~11 GB free.