feat(esh-ml1): RTX 2000E Ada on esh-pve serves embed + rerank as a LiteLLM failover
esh-pve: NVIDIA 580.178.04 (open modules, DKMS) installed on the host from NVIDIA's .run and loaded live, no reboot. nvidia-persistenced unit creates the device nodes before pve-guests; the T400's vfio-pci ids and `blacklist nvidia` retired. playbooks/esh-pve-nvidia-host.yaml. esh-ml1: CT 110, unprivileged Debian 12, 10.0.50.80, GPU nodes via devN, NVIDIA userspace from the same .run (--no-kernel-modules), docker-ce + nvidia-container-toolkit (no-cgroups). playbooks/esh-ml1-lxc.yaml. Not in the vzdump job on purpose. DNS esh-ml1.esh.internal. stacks/embed-rerank: Qwen3-Embedding-0.6B :8001 + bge-reranker-v2-m3 :8013 on the same vLLM v0.24.0 digest and flags as fv-ml1. Measured parity: embed cosine FV-vs-ESH median 0.999908 (min 0.999772), inside both self-noise floors; rerank max |delta| 0.000145 vs floor 0.000181, identical ranking. litellm: qwen3-embedding and reranker gain an esh-ml1 deployment at order 2 behind fv-ml1 (order 1). Order fallback proven with throwaway groups: refused primary +0.15 s, host-down primary ~18.7 s per call, dead-only 500. Also: repaired the DB-only alias reranker-a3-bge-v2-m3, dead since the fv-ml1 relocation (still named 10.250.50.54); documented the third unkillable homepage wedge on esh-docker-vm.
This commit is contained in:
@@ -0,0 +1,62 @@
|
||||
# `[2026-09-24]` esh-ml1 built: RTX 2000E Ada serving embed + rerank, LiteLLM order-2 failover behind fv-ml1
|
||||
|
||||
Executes Prime's 2026-09-24 decision (LXC + host driver, not a VFIO VM). Done
|
||||
2204–2240 PT the same night, **with no esh-pve reboot**.
|
||||
|
||||
**Host (esh-pve).** NVIDIA **580.178.04** open modules via DKMS from NVIDIA's
|
||||
`-no-compat32.run` (sha256 checked against NVIDIA's published sum), applied
|
||||
live by `playbooks/esh-pve-nvidia-host.yaml`: nouveau was never loaded and the
|
||||
T400's vfio ids matched nothing, so nothing held the card. Our own
|
||||
`nvidia-persistenced.service` creates `/dev/nvidia{0,ctl,-uvm,-uvm-tools}`
|
||||
**before `pve-guests`** (else `pct start` refuses the `devN:` entries). Retired
|
||||
`vfio.conf` (moved to `/root/nvidia/`) and `blacklist nvidia`. Initramfs NOT
|
||||
rebuilt on purpose (boot path untouched). 580 because the fleet's vLLM
|
||||
`v0.24.0` image is CUDA 13.0 (driver ≥ 580); fv-ml1 runs 580.65.06. `.run`
|
||||
not an apt repo: one file on both sides makes the host-module/container-lib
|
||||
version match true by construction; apt would let an upgrade move one side.
|
||||
|
||||
**CT 110 `esh-ml1`** (`playbooks/esh-ml1-lxc.yaml`): unprivileged Debian 12,
|
||||
nesting+keyctl, 6c/16G/80G `local-lvm`, `10.0.50.80` VLAN 50 (static; UDM pool
|
||||
is `.150–.250`), `startup order=30` after esh-scale/VMs. NVIDIA userspace from
|
||||
the same `.run` with `--no-kernel-modules`; docker-ce; nvidia-container-toolkit
|
||||
with `no-cgroups=true`. **Docker-in-unprivileged-LXC with the GPU worked first
|
||||
try on lxc-pve 6.0.0-2** (no AppArmor workaround needed). Fleet ids per
|
||||
`docs/pfi/fleet-conventions.md` (infra-ops 850, docker 851, vh 1000 — vh has
|
||||
no key/password yet). **Not in the vzdump job, on purpose** (rebuildable).
|
||||
DNS `esh-ml1.esh.internal` synced to all three resolvers.
|
||||
|
||||
**Stack `stacks/embed-rerank`:** `vllm-embed` :8001 + `vllm-rerank-bge` :8013,
|
||||
same models/flags/vLLM digest (`251eba5cc7c1`) as fv-ml1, 0.20 mem-util each →
|
||||
4,823 MiB used of 16,380.
|
||||
|
||||
**Parity (the load-bearing measurement; embeddings are model-specific).** 11
|
||||
texts × 2 runs per site. Embed cosine FV-vs-ESH median 0.999908 / min 0.999772,
|
||||
inside both self-noise floors (FV-vs-FV 0.999927 / 0.999791; ESH-vs-ESH
|
||||
0.999911 / 0.999809). Negative control (different texts) 0.07–0.38. Sensitivity
|
||||
floor ~2×10⁻⁴ cosine. Rerank |score| max 0.000145 vs FV-FV floor 0.000181,
|
||||
identical ranking.
|
||||
|
||||
**Gateway.** `qwen3-embedding` and `reranker` each got a second deployment
|
||||
(`order: 2`, esh-ml1); fv-ml1 is `order: 1`. LiteLLM v1.97 order-fallback
|
||||
proven with throwaway groups (created + deleted): primary refusing → ESH in
|
||||
+0.15 s (3/3); primary host-down (no ARP) → ESH but **~18.7 s on every call,
|
||||
no cooldown** (3/3); dead-only negative control → 500 (3/3). Post-restart
|
||||
production traffic is served by fv-ml1 (header `x-litellm-model-api-base`).
|
||||
`/health?model=` returned 503 with empty lists for both groups — not
|
||||
investigated. Gateway restart: liveliness back in ~52 s.
|
||||
|
||||
**Found + fixed:** DB-only alias `reranker-a3-bge-v2-m3` pointed at ana-ml2's
|
||||
pre-relocation IP `10.250.50.54` — dead since 2026-09-12 (500 after ~23 s), no
|
||||
callers in 7-day spend logs. PATCHed to `10.251.50.54:8013`. Sweep of
|
||||
`/model/info` found no other `10.250.50.54` targets.
|
||||
|
||||
**Who actually uses these names (spend logs, 7 days to 2026-09-25 05:35Z):**
|
||||
`qwen3-embedding` — worldtree-gateway 481 (nh3-dev + corviduo-dev), nevermore
|
||||
20; `reranker` — nevermore 13. **No ESH-side consumer called either in 7 days**
|
||||
(Open WebUI's RAG embeds only on document upload). So today esh-ml1's value is
|
||||
an FV-outage failover for Worldtree + nevermore, not local service for ESH.
|
||||
|
||||
**Open:** a direct ESH→esh-ml1 path (survives a mesh outage) would help only
|
||||
ESH consumers, which are idle — recommended parked. esh-ml1 not yet in
|
||||
Homepage docker.yaml or Beszel. The ~18.7 s host-down failover penalty is
|
||||
untuned (router retries/cooldown).
|
||||
Reference in New Issue
Block a user