feat(esh-ml1): RTX 2000E Ada on esh-pve serves embed + rerank as a LiteLLM failover

esh-pve: NVIDIA 580.178.04 (open modules, DKMS) installed on the host from
NVIDIA's .run and loaded live, no reboot. nvidia-persistenced unit creates the
device nodes before pve-guests; the T400's vfio-pci ids and `blacklist nvidia`
retired. playbooks/esh-pve-nvidia-host.yaml.

esh-ml1: CT 110, unprivileged Debian 12, 10.0.50.80, GPU nodes via devN,
NVIDIA userspace from the same .run (--no-kernel-modules), docker-ce +
nvidia-container-toolkit (no-cgroups). playbooks/esh-ml1-lxc.yaml. Not in the
vzdump job on purpose. DNS esh-ml1.esh.internal.

stacks/embed-rerank: Qwen3-Embedding-0.6B :8001 + bge-reranker-v2-m3 :8013 on
the same vLLM v0.24.0 digest and flags as fv-ml1. Measured parity: embed
cosine FV-vs-ESH median 0.999908 (min 0.999772), inside both self-noise
floors; rerank max |delta| 0.000145 vs floor 0.000181, identical ranking.

litellm: qwen3-embedding and reranker gain an esh-ml1 deployment at order 2
behind fv-ml1 (order 1). Order fallback proven with throwaway groups:
refused primary +0.15 s, host-down primary ~18.7 s per call, dead-only 500.

Also: repaired the DB-only alias reranker-a3-bge-v2-m3, dead since the
fv-ml1 relocation (still named 10.250.50.54); documented the third
unkillable homepage wedge on esh-docker-vm.

CLAUDE.md

@@ -362,6 +362,7 @@ Observed and standardized across servers:
 | esh-pve | 10.0.250.35 | ESH home lab | Proxmox VE hypervisor | `servers/esh-pve/README.md` |
 | esh-pve-nas | 10.0.50.55 | ESH home lab | Proxmox VE hypervisor (storage / media) | `servers/esh-pve-nas/README.md` |
 | esh-vm-db | 10.0.50.60 | ESH home lab | DB VM — PostgreSQL (paperless-ng) + MongoDB; bare-metal VM, no Docker | `servers/esh-vm-db/README.md` |
+| esh-ml1 | 10.0.50.80 | ESH home lab | GPU LXC (CT 110 on esh-pve, RTX 2000E Ada 16 GB, host-side NVIDIA driver) — embedding + reranking, LiteLLM order-2 failover behind fv-ml1 | `servers/esh-ml1/README.md` |
 
 **Placement rules:**
 - GPU-required stacks → `fv-ml1` (primary, Fountain Valley) or `irv-ml1`

dns/internal.yaml

@@ -99,6 +99,7 @@ hosts:
   - {name: esh-pve,          site: esh, v4: 10.0.250.35,    note: Proxmox hypervisor}
   - {name: esh-pve-nas,      site: esh, v4: 10.0.50.55,     note: Proxmox hypervisor — storage/media}
   - {name: esh-vm-db,        site: esh, v4: 10.0.50.60,     note: PostgreSQL + MongoDB}
+  - {name: esh-ml1,          site: esh, v4: 10.0.50.80,     note: CT110 on esh-pve — RTX 2000E Ada, embedding + reranking (vLLM)}
   - {name: vm-esh-nas,       site: esh, v4: 10.0.50.154,    note: NAS-adjacent docker host}
   - {name: esh-filebot,      site: esh, v4: 10.0.50.70,     note: restic / file-sync VM}
   - {name: esh-gw,           site: esh, v4: 10.0.250.1,     note: esh-gw}

docs/pfi/reranker-selection-ledger.md

@@ -228,3 +228,18 @@ on fv-ml1; `/model/delete` the two aliases (DB-persisted); weights retained in H
   (recreate = brief blip), (c) run a 2nd A3 replica load-balanced behind `reranker` (~2× tput,
   identical replicas so no split-measurement issue now the bake-off is closed). Brokkr will
   re-run the grid against A4 if it bites — no intuition swaps.
+
+## 2026-09-24 — second site + a dead alias repaired (infra-ops)
+
+- **`reranker-a3-bge-v2-m3` was dead from the fv-ml1 relocation (2026-09-12) until
+  2026-09-24.** It is a DB-only alias (added via /model/new, so not in `config.yaml`) and
+  still named ana-ml2's old address `10.250.50.54:8013`; every call returned 500 after ~23 s.
+  Spend logs show **no callers in the 7-day retention window** besides the probe that found
+  it. Repointed with `PATCH /model/{id}/update` to `10.251.50.54:8013` (3/3 × 200). Lesson:
+  a DB-only alias is invisible to a grep of the config, so an IP migration misses it —
+  sweep `/model/info` too.
+- **esh-ml1 added as an order-2 failover behind `reranker`** (and `qwen3-embedding`) — same
+  model and vLLM version on an RTX 2000E Ada at ESH. This is lever (c) above in failover
+  form, not load-balancing: fv-ml1 serves every request while it is up. Parity: |score|
+  FV-vs-ESH max 0.000145 vs FV-vs-FV floor 0.000181, identical ranking over 11 documents.
+  See `servers/esh-ml1/README.md`.

persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md

@@ -0,0 +1,62 @@
+# `[2026-09-24]` esh-ml1 built: RTX 2000E Ada serving embed + rerank, LiteLLM order-2 failover behind fv-ml1
+
+Executes Prime's 2026-09-24 decision (LXC + host driver, not a VFIO VM). Done
+2204–2240 PT the same night, **with no esh-pve reboot**.
+
+**Host (esh-pve).** NVIDIA **580.178.04** open modules via DKMS from NVIDIA's
+`-no-compat32.run` (sha256 checked against NVIDIA's published sum), applied
+live by `playbooks/esh-pve-nvidia-host.yaml`: nouveau was never loaded and the
+T400's vfio ids matched nothing, so nothing held the card. Our own
+`nvidia-persistenced.service` creates `/dev/nvidia{0,ctl,-uvm,-uvm-tools}`
+**before `pve-guests`** (else `pct start` refuses the `devN:` entries). Retired
+`vfio.conf` (moved to `/root/nvidia/`) and `blacklist nvidia`. Initramfs NOT
+rebuilt on purpose (boot path untouched). 580 because the fleet's vLLM
+`v0.24.0` image is CUDA 13.0 (driver ≥ 580); fv-ml1 runs 580.65.06. `.run`
+not an apt repo: one file on both sides makes the host-module/container-lib
+version match true by construction; apt would let an upgrade move one side.
+
+**CT 110 `esh-ml1`** (`playbooks/esh-ml1-lxc.yaml`): unprivileged Debian 12,
+nesting+keyctl, 6c/16G/80G `local-lvm`, `10.0.50.80` VLAN 50 (static; UDM pool
+is `.150–.250`), `startup order=30` after esh-scale/VMs. NVIDIA userspace from
+the same `.run` with `--no-kernel-modules`; docker-ce; nvidia-container-toolkit
+with `no-cgroups=true`. **Docker-in-unprivileged-LXC with the GPU worked first
+try on lxc-pve 6.0.0-2** (no AppArmor workaround needed). Fleet ids per
+`docs/pfi/fleet-conventions.md` (infra-ops 850, docker 851, vh 1000 — vh has
+no key/password yet). **Not in the vzdump job, on purpose** (rebuildable).
+DNS `esh-ml1.esh.internal` synced to all three resolvers.
+
+**Stack `stacks/embed-rerank`:** `vllm-embed` :8001 + `vllm-rerank-bge` :8013,
+same models/flags/vLLM digest (`251eba5cc7c1`) as fv-ml1, 0.20 mem-util each →
+4,823 MiB used of 16,380.
+
+**Parity (the load-bearing measurement; embeddings are model-specific).** 11
+texts × 2 runs per site. Embed cosine FV-vs-ESH median 0.999908 / min 0.999772,
+inside both self-noise floors (FV-vs-FV 0.999927 / 0.999791; ESH-vs-ESH
+0.999911 / 0.999809). Negative control (different texts) 0.07–0.38. Sensitivity
+floor ~2×10⁻⁴ cosine. Rerank |score| max 0.000145 vs FV-FV floor 0.000181,
+identical ranking.
+
+**Gateway.** `qwen3-embedding` and `reranker` each got a second deployment
+(`order: 2`, esh-ml1); fv-ml1 is `order: 1`. LiteLLM v1.97 order-fallback
+proven with throwaway groups (created + deleted): primary refusing → ESH in
++0.15 s (3/3); primary host-down (no ARP) → ESH but **~18.7 s on every call,
+no cooldown** (3/3); dead-only negative control → 500 (3/3). Post-restart
+production traffic is served by fv-ml1 (header `x-litellm-model-api-base`).
+`/health?model=` returned 503 with empty lists for both groups — not
+investigated. Gateway restart: liveliness back in ~52 s.
+
+**Found + fixed:** DB-only alias `reranker-a3-bge-v2-m3` pointed at ana-ml2's
+pre-relocation IP `10.250.50.54` — dead since 2026-09-12 (500 after ~23 s), no
+callers in 7-day spend logs. PATCHed to `10.251.50.54:8013`. Sweep of
+`/model/info` found no other `10.250.50.54` targets.
+
+**Who actually uses these names (spend logs, 7 days to 2026-09-25 05:35Z):**
+`qwen3-embedding` — worldtree-gateway 481 (nh3-dev + corviduo-dev), nevermore
+20; `reranker` — nevermore 13. **No ESH-side consumer called either in 7 days**
+(Open WebUI's RAG embeds only on document upload). So today esh-ml1's value is
+an FV-outage failover for Worldtree + nevermore, not local service for ESH.
+
+**Open:** a direct ESH→esh-ml1 path (survives a mesh outage) would help only
+ESH consumers, which are idle — recommended parked. esh-ml1 not yet in
+Homepage docker.yaml or Beszel. The ~18.7 s host-down failover penalty is
+untuned (router retries/cooldown).

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-09-24 ~2150 PT (NH3 power outage recovered: pbs-nh3 onboot set, NFS → automount. esh-pve: VM 102 retired, a 14-day hung VFIO process cleared, T400 → RTX 2000 Ada; ⭐ NEXT = provision it as LXC + host driver for embed/rerank. pfi-gx10 AC-restore patch applied but UNVALIDATED — box OFF until Prime's AC pull 09-25. elway root:root fix + fleet ownership audit. Miranda standing order + Prime callsign in CLAUDE.md. task-board mothballed.)_
+_Last updated: 2026-09-24 ~2245 PT (⭐ esh-ml1 BUILT: RTX 2000E Ada as CT 110 on esh-pve, host driver 580.178.04 live-loaded with NO reboot, vLLM embed+rerank at parity with fv-ml1, LiteLLM order-2 failover. Dead `reranker-a3-bge-v2-m3` alias repaired. ⚠ homepage wedged unkillably on esh-docker-vm (3rd time) — needs a VM reboot, Prime's call. pfi-gx10 AC-restore still UNVALIDATED — AC pull 09-25.)_
 
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -115,30 +115,26 @@ no longer deployed sidecars here. See Recent decisions.)
 
 ## Current state / in-flight
 
-_As of 2026-09-24 ~2150 PT._
+_As of 2026-09-24 ~2245 PT._
 
-### ⭐ NEXT: provision the RTX 2000 Ada on esh-pve as an LXC + host NVIDIA driver
+### ✅ esh-ml1 built (2026-09-24) — two follow-ups
 
-Prime's decision (2026-09-24, over a VFIO VM): the card serves **embedding +
-reranking offload**. Installed tonight in esh-pve's single slot (`01:00.0`,
-`10de:28b0`), no driver bound. Plan shape, not yet started:
+The RTX 2000E Ada now serves `qwen3-embedding` + `reranker` from **CT 110
+`esh-ml1`** (10.0.50.80) as the **order-2 failover** behind fv-ml1. Host driver
+went on live, no esh-pve reboot. Full record → Recent decisions.
+- **Parked unless Prime wants it:** a direct ESH→esh-ml1 path for ESH consumers
+  (survives a mesh outage). Spend logs show NO ESH-side caller in 7 days; the
+  real users are worldtree-gateway + nevermore.
+- **Not wired yet:** esh-ml1 in Homepage `docker.yaml` (needs dockerd tcp/2375)
+  and Beszel. The ~18.7 s host-down failover penalty is untuned.
 
-1. NVIDIA driver on the PVE host (headers for the running `6.8.12-*-pve`
-   kernel; DKMS). Blacklist nouveau. Remove the stale T400 vfio ids from
-   `/etc/modprobe.d` (`10de:1ff2,10de:10fa`).
-2. An LXC (Debian 12 template — PVE here rejects Debian 13) with the GPU
-   device nodes bound in and the SAME userspace driver version as the host;
-   docker + nvidia-container-toolkit inside.
-3. Serve the SAME models the fleet already uses so vectors stay compatible:
-   `qwen3-embedding` (currently fv-ml1:8001 via LiteLLM) and the rerankers
-   (`reranker`, `reranker-a3-bge-v2-m3`). Confirm sizes from the live seats
-   before choosing an engine (TEI vs vLLM).
-4. Wire as a LiteLLM failover/local deployment under the SAME model names; ESH
-   consumers (Open WebUI, Paperless) keep working if FV or the mesh is down.
+### ⚠ homepage wedged on esh-docker-vm — needs a VM reboot (Prime's call)
 
-⚠ esh-pve is ESH's only DNS and its only mesh route (esh-scale CT 108 lives
-there). Driver work means reboots: do them when an ESH outage is acceptable,
-and confirm power-off by the light, not by ping (my path in dies with esh-scale).
+3rd unkillable wedge (06-03, 09-18, now 2026-09-24 ~2224 PT): node thread D in
+`vm_mmap_pgoff`, then `exit_mmap` after a kill; `docker restart` failed. Rest
+of the VM healthy. The reboot blips ESH DNS. Signature + safe diagnosis
+commands: `servers/esh-docker-vm/README.md`. Option to move Homepage to
+ana-docker surfaced to Prime.
 
 ### pfi-gx10 is OFF — AC-pull test 2026-09-25 (Prime)
 
@@ -163,9 +159,6 @@ a shutdown (stays off by design). Outcomes and revert in
 - **infra-hermes owns a daily 0110 job** proving the first High Seat report
   (`~/.high-seat/reports/*.jsonl`) lands in a nh3-dev restic snapshot; it
   replies to svos-dev (thread `01M37P61Q85KWDVYN0A00P8856`) and pings me.
-- **Credentials in auto-memory:** the new global rule says never write one into
-  a memory file (memory is copied off-box hourly to `vh/claude-memory`). Four
-  of my memory files still carry the shared LiteLLM key literal — scrub them.
 - **Auto-memory `MEMORY.md` is over its 24.4 KB load limit** (tail truncated at
   load) — shorten index lines.
 - ESH has a single outside route (esh-scale on esh-pve) — noted, untracked.
@@ -173,6 +166,7 @@ a shutdown (stays off by design). Outcomes and revert in
 
 ## Recent decisions
 
+- `[2026-09-24]` **esh-ml1 built — RTX 2000E Ada as CT 110 on esh-pve (host driver 580.178.04 DKMS, loaded live, no reboot), vLLM embed+rerank at measured parity with fv-ml1, wired as LiteLLM `order: 2` failover; dead DB alias `reranker-a3-bge-v2-m3` (pre-relocation IP) repaired.** → `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md`
 - `[2026-09-24]` **esh-pve: VM 102 retired, a 14-day hung VFIO process found and cleared, T400 → RTX 2000 Ada; Prime chose LXC + host driver for embed/rerank** (implementation deferred to next session, tracked here + `servers/esh-pve/README.md`). → `persistent-memory.d/2026-09-24-esh-pve-vm102-retired-gpu-swap.md`
 - `[2026-09-24]` **pfi-gx10 AC-restore UEFI patch applied but NOT validated; box OFF until Prime's AC pull 2026-09-25** (tracking: `servers/pfi-gx10/README.md`, `e5197a3`). → `persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md`
 - `[2026-09-24]` **NH3 power outage recovered** — pbs-nh3 had no `onboot` (set), NFS boot race fixed with automount (`1cbde50`), every other Claude session on nh3-dev died. → `persistent-memory.d/2026-09-24-nh3-power-outage-recovery.md`

playbooks/esh-ml1-lxc.yaml

@@ -0,0 +1,175 @@
+# esh-ml1 — the GPU LXC on esh-pve that serves embedding + reranking on the
+# RTX 2000E Ada (16 GB). Runs against the HYPERVISOR and drives the container
+# through `pct`, so it works before the container has any users or keys.
+#
+# Prerequisite: playbooks/esh-pve-nvidia-host.yaml (host kernel module, device
+# nodes, persistence unit ordered before pve-guests).
+#
+# WHAT IT BUILDS
+#   - CT {{ ctid }} `esh-ml1`, unprivileged, Debian 12 (the template the host's
+#     own bookworm matches), nesting+keyctl for Docker, static 10.0.50.80 on
+#     VLAN 50 (outside the UDM's .150–.250 DHCP pool), onboot AFTER esh-scale
+#     and the two VMs so a GPU fault can never delay DNS or the mesh route.
+#   - /dev/nvidia0, nvidiactl, nvidia-uvm, nvidia-uvm-tools bound in via
+#     `devN:` (PVE handles the cgroup rules and node creation).
+#   - NVIDIA userspace from THE SAME .run the host installed, with
+#     --no-kernel-modules. Host module and container libraries must be the same
+#     version or every CUDA call fails with "driver/library version mismatch".
+#   - Fleet identities per docs/pfi/fleet-conventions.md: docker gid 851
+#     (created BEFORE docker-ce so the package does not pick one), infra-ops
+#     850 NOPASSWD, vh 1000, /opt/docker root:docker 2775.
+#   - docker-ce + nvidia-container-toolkit with no-cgroups=true (an unprivileged
+#     LXC cannot write the device cgroup; the LXC's own devN rules already
+#     confine the GPU).
+#
+# NOT BACKED UP, on purpose: esh-pve's vzdump job lists vmids explicitly and
+# this CT is not in it. Everything here is rebuilt by this playbook plus
+# stacks/embed-rerank; the models re-download from Hugging Face.
+#
+# Run:  scripts/elway root@esh-pve --playbook playbooks/esh-ml1-lxc.yaml
+
+vars:
+  ctid: 110
+  hostname: esh-ml1
+  ip_cidr: 10.0.50.80/24
+  gateway: 10.0.50.1
+  vlan: 50
+  cores: 6
+  memory_mb: 16384
+  swap_mb: 2048
+  rootfs_gb: 80
+  template: debian-12-standard_12.12-1_amd64.tar.zst
+  # Must equal playbooks/esh-pve-nvidia-host.yaml's driver_version.
+  driver_version: 580.178.04
+  stage_dir: /root/nvidia
+  infra_ops_pubkey: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN+1HBwfXrkfTYWdcnWCjLJ6VLAGC87gxH5h5vKaaA3c infra-ops@pfi-fleet"
+
+steps:
+  - name: Fetch the Debian 12 template
+    shell: pveam update >/dev/null && pveam download local {{ template }}
+    creates: /var/lib/vz/template/cache/{{ template }}
+
+  - name: Create CT {{ ctid }} ({{ hostname }}) with the GPU device nodes bound in
+    shell: |
+      pct create {{ ctid }} local:vztmpl/{{ template }} \
+        --hostname {{ hostname }} --unprivileged 1 --features nesting=1,keyctl=1 \
+        --cores {{ cores }} --memory {{ memory_mb }} --swap {{ swap_mb }} \
+        --rootfs local-lvm:{{ rootfs_gb }} \
+        --net0 name=eth0,bridge=vmbr0,firewall=1,gw={{ gateway }},ip={{ ip_cidr }},tag={{ vlan }},type=veth \
+        --onboot 1 --startup order=30,up=10 \
+        --dev0 /dev/nvidia0 --dev1 /dev/nvidiactl \
+        --dev2 /dev/nvidia-uvm --dev3 /dev/nvidia-uvm-tools \
+        --description "esh-ml1 — RTX 2000E Ada embedding + reranking (vLLM). Built by eshpfi playbooks/esh-ml1-lxc.yaml; stack stacks/embed-rerank. NOT in the vzdump job on purpose (rebuildable)."
+    when: "! pct status {{ ctid }} >/dev/null 2>&1"
+
+  - name: Start the container
+    shell: pct start {{ ctid }} && sleep 5
+    when: "! pct status {{ ctid }} | grep -q running"
+
+  - name: Base packages + bookworm point upgrade
+    shell: |
+      pct exec {{ ctid }} -- bash -s <<'EOF'
+      set -euo pipefail
+      export DEBIAN_FRONTEND=noninteractive
+      for i in $(seq 1 30); do getent hosts deb.debian.org >/dev/null && break; sleep 1; done
+      apt-get update -qq
+      apt-get -y -qq full-upgrade
+      apt-get install -y -qq --no-install-recommends ca-certificates curl gnupg sudo kmod jq less rsync locales
+      EOF
+    # rsync: scripts/deploy-stack.sh pushes stacks with it.
+    when: "! pct exec {{ ctid }} -- sh -c 'command -v sudo && command -v jq && command -v rsync && command -v locale-gen' >/dev/null 2>&1"
+
+  - name: Generate the en_US.UTF-8 locale the template's LANG names
+    # The Debian template sets LANG=en_US.UTF-8 but ships no locale, so every
+    # apt and perl run warns about it.
+    shell: |
+      pct exec {{ ctid }} -- bash -s <<'EOF'
+      set -euo pipefail
+      sed -i 's/^# *en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen
+      locale-gen >/dev/null
+      EOF
+    when: "! pct exec {{ ctid }} -- locale -a 2>/dev/null | grep -qi '^en_US.utf8$'"
+
+  - name: Fleet identities (docker 851, infra-ops 850, vh 1000) + /opt/docker tree
+    shell: |
+      pct exec {{ ctid }} -- bash -s <<'EOF'
+      set -euo pipefail
+      getent group docker >/dev/null || groupadd -g 851 docker
+      getent group infra-ops >/dev/null || groupadd -g 850 infra-ops
+      id infra-ops >/dev/null 2>&1 || useradd -u 850 -g 850 -G docker -m -s /bin/bash infra-ops
+      chmod 0700 /home/infra-ops
+      install -d -m 0700 -o infra-ops -g infra-ops /home/infra-ops/.ssh
+      echo '{{ infra_ops_pubkey }}' > /home/infra-ops/.ssh/authorized_keys
+      chown infra-ops:infra-ops /home/infra-ops/.ssh/authorized_keys
+      chmod 0600 /home/infra-ops/.ssh/authorized_keys
+      echo 'infra-ops ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/infra-ops
+      chmod 0440 /etc/sudoers.d/infra-ops
+      id vh >/dev/null 2>&1 || useradd -u 1000 -U -G docker,sudo -m -s /bin/bash vh
+      chmod 0700 /home/vh
+      install -d -m 2775 -o root -g docker /opt/docker /opt/docker/compose /opt/docker/conf
+      EOF
+    when: "! pct exec {{ ctid }} -- sh -c 'test \"$(id -u infra-ops)\" = 850 && test \"$(getent group docker | cut -d: -f3)\" = 851 && test -d /opt/docker/compose'"
+
+  - name: Hugging Face cache for the embed-rerank stack
+    shell: pct exec {{ ctid }} -- install -d -m 2775 -o root -g docker /opt/aimodels /opt/aimodels/huggingface
+    when: "! pct exec {{ ctid }} -- test -d /opt/aimodels/huggingface"
+
+  - name: NVIDIA userspace {{ driver_version }} from the host's .run (no kernel modules)
+    shell: |
+      set -e
+      f=NVIDIA-Linux-x86_64-{{ driver_version }}-no-compat32.run
+      pct push {{ ctid }} {{ stage_dir }}/$f /root/$f --perms 0755
+      pct exec {{ ctid }} -- sh /root/$f --no-kernel-modules --silent --no-x-check --no-opengl-files
+      pct exec {{ ctid }} -- rm -f /root/$f
+    when: "! pct exec {{ ctid }} -- sh -c 'nvidia-smi --query-gpu=driver_version --format=csv,noheader 2>/dev/null' | grep -qx '{{ driver_version }}'"
+
+  - name: docker-ce from Docker's apt repo
+    shell: |
+      pct exec {{ ctid }} -- bash -s <<'EOF'
+      set -euo pipefail
+      export DEBIAN_FRONTEND=noninteractive
+      install -m 0755 -d /etc/apt/keyrings
+      curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
+      chmod a+r /etc/apt/keyrings/docker.asc
+      echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian bookworm stable" \
+        > /etc/apt/sources.list.d/docker.list
+      apt-get update -qq
+      apt-get install -y -qq docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
+      EOF
+    when: "! pct exec {{ ctid }} -- sh -c 'command -v docker' >/dev/null 2>&1"
+
+  - name: nvidia-container-toolkit, wired into dockerd, no-cgroups for the LXC
+    shell: |
+      pct exec {{ ctid }} -- bash -s <<'EOF'
+      set -euo pipefail
+      export DEBIAN_FRONTEND=noninteractive
+      curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
+        | gpg --dearmor --yes -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
+      curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
+        | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
+        > /etc/apt/sources.list.d/nvidia-container-toolkit.list
+      apt-get update -qq
+      apt-get install -y -qq nvidia-container-toolkit
+      nvidia-ctk runtime configure --runtime=docker
+      nvidia-ctk config --set nvidia-container-cli.no-cgroups=true --in-place
+      systemctl restart docker
+      EOF
+    when: "! pct exec {{ ctid }} -- sh -c 'grep -q nvidia /etc/docker/daemon.json && grep -Eq \"^no-cgroups *= *true\" /etc/nvidia-container-runtime/config.toml' 2>/dev/null"
+
+verify:
+  - name: Container is running with onboot set, started after the core guests
+    shell: "pct status {{ ctid }} | grep -q running && pct config {{ ctid }} | grep -q '^onboot: 1' && pct config {{ ctid }} | grep -q '^startup: order=30'"
+    changed_when: "false"
+
+  - name: nvidia-smi inside the CT reports the host's driver version
+    shell: pct exec {{ ctid }} -- nvidia-smi --query-gpu=name,driver_version --format=csv,noheader | grep -q 'RTX 2000.*, {{ driver_version }}$'
+    changed_when: "false"
+
+  - name: A Docker container gets the GPU (toolkit + no-cgroups work end to end)
+    shell: pct exec {{ ctid }} -- docker run --rm --gpus all debian:12-slim nvidia-smi -L | grep -q 'RTX 2000'
+    changed_when: "false"
+
+  - name: Fleet identities are the pinned ids
+    shell: |
+      pct exec {{ ctid }} -- sh -c 'test "$(id -u infra-ops)" = 850 && test "$(id -u vh)" = 1000 && test "$(getent group docker | cut -d: -f3)" = 851 && id -nG infra-ops | grep -qw docker'
+    changed_when: "false"

playbooks/esh-pve-nvidia-host.yaml

@@ -0,0 +1,137 @@
+# esh-pve — NVIDIA driver on the HYPERVISOR, for the RTX 2000 Ada that an LXC
+# (esh-ml1, embedding + reranking) uses.
+#
+# WHY A HOST DRIVER: Prime's decision 2026-09-24 — LXC + host driver, NOT a
+# VFIO VM. Both esh-pve hangs this year (the 2026-08-19 hard freeze and the
+# 2026-09-10 vzdump that left `kvm -id 102` in D state for 14 days) came from
+# VFIO passthrough on this box. An LXC shares the host kernel's driver, so no
+# passthrough, no pinned VM RAM. Accepted cost: a driver on the hypervisor,
+# rebuilt by DKMS on every PVE kernel update, and the LXC's userspace libraries
+# must be the SAME version as this kernel module (see
+# playbooks/esh-ml1-lxc.yaml, which installs from the same .run file).
+#
+# WHY THE .run FILE, NOT AN APT REPO: the LXC must match the host module
+# exactly. One file, installed with the kernel module here and with
+# --no-kernel-module in the container, makes the match true by construction.
+# An apt repo on both sides would let a routine `apt upgrade` move one side
+# and not the other, which breaks every CUDA call in the container.
+#
+# WHY 580: the fleet's vLLM image (vllm/vllm-openai:v0.24.0) is a CUDA 13.0
+# build and needs a >= 580 driver. fv-ml1 runs 580.65.06; this is the newest
+# point release on the same branch. `-no-compat32` because a headless compute
+# box has no use for 32-bit libraries.
+#
+# NO REBOOT NEEDED: nouveau is blacklisted and was never loaded, and the T400's
+# vfio-pci ids no longer match any device, so nothing holds the card. The
+# module loads live. (esh-pve is ESH's only DNS and mesh route; a reboot takes
+# the whole site dark from outside — avoid one.)
+#
+# Upgrading the driver: bump driver_version + driver_sha256 here AND in
+# playbooks/esh-ml1-lxc.yaml, run this, then that. Until both have run, CUDA
+# in the container fails with "driver/library version mismatch".
+#
+# Run:  scripts/elway root@esh-pve --playbook playbooks/esh-pve-nvidia-host.yaml
+
+vars:
+  driver_version: 580.178.04
+  # From download.nvidia.com/XFree86/Linux-x86_64/580.178.04/*.run.sha256sum
+  driver_sha256: 1833c8c5c53481dfab46df442c3de26ec5d4c35084e03d94ad93746d29899750
+  stage_dir: /root/nvidia
+
+steps:
+  - name: Stage the driver .run and check it against NVIDIA's published sha256
+    shell: |
+      set -e
+      mkdir -p {{ stage_dir }}
+      f={{ stage_dir }}/NVIDIA-Linux-x86_64-{{ driver_version }}-no-compat32.run
+      if ! echo "{{ driver_sha256 }}  $f" | sha256sum -c --status 2>/dev/null; then
+        curl -fsSL -o "$f.part" \
+          https://download.nvidia.com/XFree86/Linux-x86_64/{{ driver_version }}/NVIDIA-Linux-x86_64-{{ driver_version }}-no-compat32.run
+        echo "{{ driver_sha256 }}  $f.part" | sha256sum -c
+        mv "$f.part" "$f"
+        chmod 0755 "$f"
+      fi
+    when: "! echo '{{ driver_sha256 }}  {{ stage_dir }}/NVIDIA-Linux-x86_64-{{ driver_version }}-no-compat32.run' | sha256sum -c --status 2>/dev/null"
+
+  # The T400 was passed to VM 102 through vfio-pci by id. The card and the VM
+  # are both gone; the ids match nothing. Kept (not deleted) so the retirement
+  # is readable later.
+  - name: Retire the T400's vfio-pci id binding
+    shell: mv /etc/modprobe.d/vfio.conf {{ stage_dir }}/vfio.conf.retired-2026-09-24
+    when: "grep -q '10de:1ff2' /etc/modprobe.d/vfio.conf 2>/dev/null"
+
+  # blacklist.conf blacklisted `nvidia` alongside nouveau (VM-passthrough era,
+  # when the host must never touch the card). nouveau stays blacklisted.
+  - name: Stop blacklisting the nvidia module (nouveau stays blacklisted)
+    shell: |
+      cp -n /etc/modprobe.d/blacklist.conf {{ stage_dir }}/blacklist.conf.pre-nvidia
+      sed -i '/^blacklist nvidia$/d' /etc/modprobe.d/blacklist.conf
+    when: "grep -qx 'blacklist nvidia' /etc/modprobe.d/blacklist.conf"
+
+  - name: Install the driver with DKMS (open kernel modules)
+    # --no-opengl-files: headless compute. --kernel-module-type=open: Ada
+    # (Turing+) is supported by the open modules and fv-ml1 runs them too.
+    # DKMS rebuilds the module whenever a new proxmox-headers package lands;
+    # `proxmox-headers-6.8` (the meta-package) is installed so new headers
+    # arrive with each new kernel.
+    shell: |
+      sh {{ stage_dir }}/NVIDIA-Linux-x86_64-{{ driver_version }}-no-compat32.run \
+        --dkms --silent --no-x-check --no-opengl-files --kernel-module-type=open \
+        --kernel-name="$(uname -r)"
+    when: "! dkms status -m nvidia -v {{ driver_version }} -k $(uname -r) 2>/dev/null | grep -q installed"
+
+  - name: Load nvidia + nvidia_uvm at every boot
+    shell: |
+      printf '# RTX 2000 Ada for the esh-ml1 LXC — see playbooks/esh-pve-nvidia-host.yaml\nnvidia\nnvidia_uvm\n' \
+        > /etc/modules-load.d/nvidia.conf
+    when: "! grep -qx 'nvidia_uvm' /etc/modules-load.d/nvidia.conf 2>/dev/null"
+
+  # The LXC binds /dev/nvidia0, /dev/nvidiactl and /dev/nvidia-uvm{,-tools}.
+  # Those nodes do not exist until something asks for them, and `pct start`
+  # refuses a dev passthrough whose host path is missing. This unit creates
+  # them and keeps the GPU initialised (persistence mode), and runs BEFORE
+  # pve-guests so onboot guests find them. If it fails, only the GPU LXC fails
+  # to start; the other guests do not depend on it.
+  - name: Install the GPU-init + persistence unit, ordered before pve-guests
+    shell: |
+      cat > /etc/systemd/system/nvidia-persistenced.service <<'EOF'
+      # See playbooks/esh-pve-nvidia-host.yaml
+      [Unit]
+      Description=NVIDIA persistence daemon + device nodes for the GPU LXC
+      Before=pve-guests.service
+      After=systemd-modules-load.service
+
+      [Service]
+      Type=forking
+      # -c0 -u: load nvidia_uvm and create /dev/nvidia-uvm{,-tools} and /dev/nvidia0.
+      ExecStartPre=/usr/bin/nvidia-modprobe -c0 -u
+      ExecStart=/usr/bin/nvidia-persistenced --user root
+      ExecStopPost=/bin/rm -rf /var/run/nvidia-persistenced
+
+      [Install]
+      WantedBy=multi-user.target
+      EOF
+      systemctl daemon-reload
+      systemctl enable --now nvidia-persistenced.service
+    when: "! systemctl is-active --quiet nvidia-persistenced.service || ! grep -q 'Before=pve-guests.service' /etc/systemd/system/nvidia-persistenced.service"
+
+verify:
+  - name: nvidia-smi sees the RTX 2000 Ada at the pinned driver version
+    shell: nvidia-smi --query-gpu=name,driver_version --format=csv,noheader | grep -q 'RTX 2000.*, {{ driver_version }}$'
+    changed_when: "false"
+
+  - name: Kernel module in use is the DKMS-built {{ driver_version }}
+    shell: grep -q '{{ driver_version }}' /proc/driver/nvidia/version && dkms status -m nvidia -v {{ driver_version }} -k $(uname -r) | grep -q installed
+    changed_when: "false"
+
+  - name: Device nodes the LXC binds all exist
+    shell: test -c /dev/nvidia0 && test -c /dev/nvidiactl && test -c /dev/nvidia-uvm && test -c /dev/nvidia-uvm-tools
+    changed_when: "false"
+
+  - name: Persistence mode is on
+    shell: nvidia-smi --query-gpu=persistence_mode --format=csv,noheader | grep -qx Enabled
+    changed_when: "false"
+
+  - name: Survives a reboot (modules-load + unit enabled before pve-guests)
+    shell: grep -qx nvidia_uvm /etc/modules-load.d/nvidia.conf && systemctl is-enabled --quiet nvidia-persistenced.service && ! grep -qx 'blacklist nvidia' /etc/modprobe.d/blacklist.conf
+    changed_when: "false"

servers/esh-docker-vm/README.md

@@ -73,3 +73,32 @@ Like `nh3-docker`, this host runs **Dozzle** and **Beszel** agents that report b
 ## Placement rule
 
 Home-lab workloads for the ESH site go here. Not part of the PFI colo topology.
+
+## ⚠ Recurring: `homepage` wedges unkillably (3× — 2026-06-03, 2026-09-18, 2026-09-24)
+
+The Homepage container stops answering (Uptime Kuma: `timeout of 16000ms
+exceeded`; healthcheck `Connecting to 127.0.0.1:3000` times out) and cannot be
+stopped. Signature captured 2026-09-24 ~2224 PT, guest kernel `6.1.0-41-amd64`:
+
+- one `node` thread in **D state in `vm_mmap_pgoff`** — waiting for its own
+  process's `mmap_lock` for write, with **no visible holder** (every other
+  thread sat in `futex_wait`; a scan of every task's kernel stack found no
+  reader in a fault, NFS or `access_remote_vm` path except the `ps` calls
+  queued behind it).
+- `docker restart` → *"tried to kill container, but did not receive an exit
+  event"*; the process then sits in **`exit_mmap`** (uninterruptible) with
+  PID 1 of the container in `zap_pid_ns_processes`. The June entry in
+  archival-memory records the same `exit_mmap` end state.
+- The rest of the VM is fine: DNS (AdGuard) answered, 17 other containers up.
+
+⚠ **Diagnosing it can hang your shell.** `ps`, `pgrep` and `docker top` read
+`/proc/<pid>/cmdline|environ`, which takes the same lock, so they block in D
+state too. Read `/proc/<pid>/task/*/stat` and `sudo cat .../stack` instead
+(`ssh infra-ops@10.0.50.45`, NOPASSWD).
+
+**Only a VM reboot clears it**, and this VM is ESH's DNS resolver, so the
+reboot is a short ESH-wide DNS outage — schedule it. Root cause not
+established; a kernel-side mmap_lock problem is the leading guess, not a
+finding. Moving Homepage to ana-docker (as was done for Uptime Kuma on
+2026-09-21 for the same box's history) would take the dashboard out of this
+failure domain.

servers/esh-ml1/README.md

@@ -0,0 +1,83 @@
+# esh-ml1
+
+GPU LXC for the ESH home lab: **CT 110 on esh-pve**, holding the **NVIDIA RTX
+2000E Ada** (16 GB, 50 W, `01:00.0`). It serves the fleet's embedding and
+reranking models locally at ESH. Built 2026-09-24.
+
+| | |
+|---|---|
+| **IP** | `10.0.50.80/24`, VLAN 50, gateway `10.0.50.1` (static, outside the UDM's `.150–.250` DHCP pool) |
+| **DNS** | `esh-ml1.esh.internal` |
+| **SSH** | `ssh esh-ml1` → `infra-ops@10.0.50.80` (NOPASSWD sudo) · from the host: `pct enter 110` |
+| **OS** | Debian 12, unprivileged, `nesting=1,keyctl=1` |
+| **Size** | 6 cores, 16 GB RAM + 2 GB swap, 80 GB rootfs on `local-lvm` |
+| **Boot** | `onboot: 1`, `startup: order=30` — after esh-scale (1), esh-vm-db (10) and esh-vm-docker (20), so a GPU fault never delays ESH's DNS or mesh route |
+| **Backups** | **None, on purpose.** esh-pve's vzdump job lists vmids explicitly and 110 is not one. Everything is rebuilt from the playbooks and the stack; models re-download. |
+
+## What it serves
+
+`stacks/embed-rerank` (`/opt/docker/compose/embed-rerank`):
+
+| container | model | port | gateway name |
+|---|---|---|---|
+| `vllm-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 | `qwen3-embedding` (order 2) |
+| `vllm-rerank-bge` | `BAAI/bge-reranker-v2-m3` | 8013 | `reranker` (order 2) |
+
+The same models, vLLM version (`v0.24.0`, digest `251eba5cc7c1`) and flags as
+fv-ml1's `vllm` stack, so the two sites are interchangeable. In LiteLLM they
+are the **order-2 failover** behind fv-ml1: fv-ml1 serves every request while
+it is up.
+
+**Parity, measured 2026-09-24** (11 texts incl. CJK, code, a 6k-char passage;
+2 runs per site):
+
+| | median | min |
+|---|---|---|
+| embed cosine FV vs ESH, same text | 0.999908 | 0.999772 |
+| noise floor FV vs FV | 0.999927 | 0.999791 |
+| noise floor ESH vs ESH | 0.999911 | 0.999809 |
+| negative control, different texts | 0.232 | 0.071 |
+
+The cross-site difference is inside each site's own run-to-run noise; this
+method cannot resolve a cosine gap below ~2×10⁻⁴. Reranker scores differed by
+at most 0.000145 (FV-vs-FV floor 0.000181), with identical ranking.
+
+**VRAM:** 0.20 × 16,380 MiB each; 4,823 MiB in use with both loaded, ~11 GB
+free.
+
+## How it is built
+
+1. [`playbooks/esh-pve-nvidia-host.yaml`](../../playbooks/esh-pve-nvidia-host.yaml)
+   — driver **580.178.04** (open modules, DKMS) on the **hypervisor**, the
+   `nvidia-persistenced` unit that creates the device nodes before
+   `pve-guests`, and removal of the old VFIO/blacklist config.
+2. [`playbooks/esh-ml1-lxc.yaml`](../../playbooks/esh-ml1-lxc.yaml) — the CT,
+   `dev0–3` GPU nodes, the NVIDIA userspace from the **same `.run`** with
+   `--no-kernel-modules`, fleet ids (infra-ops 850, docker 851, vh 1000),
+   docker-ce and nvidia-container-toolkit with `no-cgroups = true`.
+3. `scripts/deploy-stack.sh esh-ml1 embed-rerank`, then `docker compose up -d`.
+
+Both playbooks are idempotent; re-run them to repair.
+
+## ⚠ Driver version lock
+
+The kernel module lives on esh-pve; the libraries live in this container. They
+**must be the same version**, or every CUDA call fails with *"driver/library
+version mismatch"*. To upgrade: bump `driver_version` + `driver_sha256` in the
+host playbook and `driver_version` in the LXC playbook, run the host one, then
+the LXC one, then restart the stack.
+
+A PVE kernel update is handled by DKMS (`proxmox-headers-6.8` pulls headers
+for each new kernel). Moving esh-pve to a different kernel series (6.14 opt-in)
+needs that series' headers meta-package installed first, or the module will not
+build and this CT will fail to start at the next boot.
+
+## Not yet wired
+
+- **Homepage**: the compose carries labels, but esh-ml1 is not in
+  `stacks/homepage/conf/docker.yaml` (it would need dockerd on tcp/2375 like
+  the other hosts).
+- **Beszel**: no agent yet.
+- **ESH consumers** (Open WebUI RAG, Paperless) still go through the gateway
+  at ana-docker, so they do not survive a mesh outage. See persistent-memory
+  for the open decision.

servers/esh-ml1/ssh-target

@@ -0,0 +1 @@
+infra-ops@10.0.50.80

servers/esh-pve/README.md

@@ -13,7 +13,8 @@ Proxmox VE hypervisor for the ESH home lab (`pve.esteban.net`). Non-PFI scope.
 
 - **CPU:** 13th Gen Intel Core i9-13900H
 - **RAM:** 62.5 GB
-- **Kernel:** `6.8.12-16-pve` (Proxmox 8.x)
+- **Kernel:** `6.8.12-42-pve`, PVE 8.4.20 (as of 2026-09-24)
+- **GPU:** NVIDIA RTX 2000E Ada, 16 GB (`01:00.0`) — driven by the **host** (580.178.04, DKMS), used by the esh-ml1 LXC. See below.
 - **Storage:** local `pve-root` at 94G (21% used) + two large NFS mounts from `10.0.50.50`:
   - `/mnt/pve/esh-nas` — 92 TB available on `/mnt/pvestore`
   - `/mnt/pve/tank-vmbu` — 93 TB, VM backup target (~1.4 TB used)
@@ -47,10 +48,39 @@ fully up. esh-vm-docker, esh-vm-db and esh-scale all started by themselves.
 esh-vm-docker's NFS (`/mnt/books`, `/mnt/backup`) mounted, traefik served with no
 403s, and AdGuard resolved.
 
-The new card has **no driver bound**. `/etc/modprobe.d` still lists the T400's IDs
-for vfio-pci (`10de:1ff2,10de:10fa`), which no longer match anything; harmless, but
-clean it up when the card's use is decided. The intended use is a local
-embedding endpoint.
+The new card got its driver the same night — see the next section.
+
+## GPU — host driver + the esh-ml1 LXC (2026-09-24)
+
+**Prime's decision: LXC + host driver, NOT a VFIO VM.** Both hangs on this box
+this year came from VFIO passthrough. The card now runs under the host's own
+NVIDIA driver and is shared into **CT 110 `esh-ml1`**, which serves embedding
+and reranking ([`servers/esh-ml1/README.md`](../esh-ml1/README.md)).
+
+- **Driver 580.178.04, open kernel modules, DKMS**, from NVIDIA's `.run`
+  (`/root/nvidia/`). Applied by
+  [`playbooks/esh-pve-nvidia-host.yaml`](../../playbooks/esh-pve-nvidia-host.yaml)
+  **live, with no reboot**: nouveau was never loaded and nothing held the card.
+- **`nvidia-persistenced.service`** (ours, in `/etc/systemd/system`) runs
+  `nvidia-modprobe -c0 -u` and the persistence daemon **before
+  `pve-guests`**, so `/dev/nvidia0`, `nvidiactl` and `nvidia-uvm{,-tools}`
+  exist when CT 110 starts. Without them `pct start` refuses the `devN:`
+  entries. If the unit fails, only CT 110 fails; no other guest depends on it.
+- `/etc/modules-load.d/nvidia.conf` loads `nvidia` + `nvidia_uvm`.
+- **Retired:** the T400's vfio-pci id binding (`/etc/modprobe.d/vfio.conf`, moved
+  to `/root/nvidia/vfio.conf.retired-2026-09-24`) and `blacklist nvidia` in
+  `/etc/modprobe.d/blacklist.conf` (`blacklist nouveau` stays). The `vfio*`
+  lines in `/etc/modules` were left alone; they bind nothing. The initramfs was
+  not rebuilt, to keep the boot path untouched; the next kernel update will.
+- The installer also enabled NVIDIA's `nvidia-{suspend,hibernate,resume}`
+  units. They only act on a suspend and are harmless on a server.
+- ⚠ **Kernel updates:** DKMS rebuilds the module when a new
+  `proxmox-headers-6.8.*` arrives (the `proxmox-headers-6.8` meta-package is
+  installed). Before opting into a different kernel series, install that
+  series' headers meta-package, or the next boot comes up with no GPU and CT
+  110 will not start.
+- ⚠ **The host module and CT 110's libraries must stay the same version.**
+  Upgrade both playbooks together (host first).
 
 ## Watchdog — hardware, not software
 

stacks/embed-rerank/.env.example

@@ -0,0 +1,29 @@
+# embed-rerank tunables (esh-ml1). Copy to `.env` on the server.
+#
+# Everything model-shaped here MUST match fv-ml1's stacks/vllm .env: the same
+# models, the same vLLM version, the same max-model-len. A drift in the
+# embedding model or its version makes this seat's vectors incompatible with
+# every index built against fv-ml1's.
+
+# PINNED to fv-ml1's version. Bump both sites together.
+VLLM_VERSION=v0.24.0
+
+# Same host ports as fv-ml1 (container listens on 8000).
+EMBED_PORT=8001
+RERANK_PORT=8013
+
+EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B
+RERANK_MODEL=BAAI/bge-reranker-v2-m3
+
+# Fractions of the RTX 2000E Ada's 16,380 MiB. fv-ml1 runs 0.03 of a 96 GB
+# card (~2.9 GB each); 0.20 here is ~3.2 GB each — the same budget plus a
+# little, leaving ~9.5 GB free.
+EMBED_GPU_MEM_UTIL=0.20
+RERANK_GPU_MEM_UTIL=0.20
+EMBED_MAX_MODEL_LEN=8192
+RERANK_MAX_MODEL_LEN=8192
+
+# fv-ml1's seats run with no API key (LiteLLM fronts them); match that.
+API_KEY=
+# Both models are public; no token needed.
+HF_TOKEN=

stacks/embed-rerank/README.md

@@ -0,0 +1,37 @@
+# embed-rerank
+
+The fleet's embedding + reranking models on **esh-ml1** (CT 110 on esh-pve,
+RTX 2000E Ada). The second site for `qwen3-embedding` and `reranker`; fv-ml1's
+[`vllm`](../vllm/) stack is the first.
+
+| container | model | port |
+|---|---|---|
+| `vllm-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 |
+| `vllm-rerank-bge` | `BAAI/bge-reranker-v2-m3` | 8013 |
+
+**Keep it in lockstep with `stacks/vllm`:** same models, same `VLLM_VERSION`,
+same `--max-model-len`. The LiteLLM groups treat the two sites as one model,
+so any drift in the embedding model or its version silently mixes
+incompatible vectors into consumers' indexes.
+
+## Deploy
+
+```bash
+scripts/deploy-stack.sh esh-ml1 embed-rerank
+ssh esh-ml1 'cd /opt/docker/compose/embed-rerank && cp -n .env.example .env && docker compose config -q && docker compose up -d'
+```
+
+Host prerequisites (driver, LXC, docker, toolkit) are in
+[`servers/esh-ml1/README.md`](../../servers/esh-ml1/README.md).
+
+## Smoke test
+
+```bash
+curl -s http://10.0.50.80:8001/v1/embeddings -H 'content-type: application/json' \
+  -d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":"hello"}' | jq '.data[0].embedding | length'   # 1024
+curl -s http://10.0.50.80:8013/rerank -H 'content-type: application/json' \
+  -d '{"model":"BAAI/bge-reranker-v2-m3","query":"cat","documents":["a cat","a car"]}' | jq '.results[0]'
+```
+
+Parity against fv-ml1 is recorded in the host README; re-measure after any
+version bump on either side.

stacks/embed-rerank/compose.yaml

@@ -0,0 +1,130 @@
+# embed-rerank — the fleet's embedding + reranking models, served locally at ESH
+# on esh-ml1 (CT 110 on esh-pve, RTX 2000E Ada, 16 GB).
+#
+# WHY: Prime's decision 2026-09-24 — the Ada card offloads embedding and
+# reranking so ESH consumers (Open WebUI RAG, Paperless) are not tied to one
+# seat on fv-ml1. It serves the SAME models as fv-ml1's `vllm` stack, with the
+# SAME vLLM version and flags, because embedding vectors are model-specific:
+# a different embedding model here would silently poison every index built
+# against fv-ml1's. Mirror stacks/vllm/compose.yaml when that changes.
+#
+#   vllm-embed        Qwen/Qwen3-Embedding-0.6B  → /v1/embeddings   :8001
+#   vllm-rerank-bge   BAAI/bge-reranker-v2-m3    → /rerank, /score  :8013
+#
+# Ports match fv-ml1 on purpose, so a LiteLLM deployment for either site
+# differs only in the host part of api_base.
+#
+# `vllm-rerank-bge`, not fv-ml1's `vllm-rerank-a3`: that name carries R43
+# bake-off provenance for THAT container; and plain `vllm-rerank` was the
+# retired Qwen3-Reranker that measured harmful. Name the model instead.
+#
+# NO `tnet`/traefik-net: esh-ml1 runs no traefik, and an external network
+# that does not exist would stop the stack from starting. Consumers reach the
+# published ports directly.
+#
+# Host setup (driver, LXC, docker, toolkit): playbooks/esh-pve-nvidia-host.yaml
+# then playbooks/esh-ml1-lxc.yaml. Tunables live in .env — edit that, not this.
+
+name: embed-rerank
+
+services:
+  vllm-embed:
+    image: vllm/vllm-openai:${VLLM_VERSION}
+    container_name: vllm-embed
+    restart: unless-stopped
+    ipc: host
+    ports:
+      - "${EMBED_PORT}:8000"
+    volumes:
+      - /opt/aimodels/huggingface:/hfcache
+    environment:
+      - HF_HOME=/hfcache
+      - HF_HUB_CACHE=/hfcache/hub
+      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
+      - VLLM_API_KEY=${API_KEY:-}
+    command:
+      - ${EMBED_MODEL}
+      - --served-model-name
+      - ${EMBED_MODEL}
+      - --runner
+      - pooling
+      - --host
+      - 0.0.0.0
+      - --port
+      - "8000"
+      - --gpu-memory-utilization
+      - ${EMBED_GPU_MEM_UTIL}
+      - --max-model-len
+      - ${EMBED_MAX_MODEL_LEN}
+      - --dtype
+      - auto
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              device_ids: ["0"]
+              capabilities: [gpu]
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 180s
+    labels:
+      - homepage.group=AI - Eval & Retrieval
+      - homepage.name=vLLM Embed (esh-ml1)
+      - homepage.icon=mdi-vector-arrange-below
+      - homepage.description=Qwen3 Embedding 0.6B via vLLM (esh-ml1, RTX 2000E Ada)
+      - homepage.href=http://10.0.50.80:${EMBED_PORT}/docs
+
+  vllm-rerank-bge:
+    image: vllm/vllm-openai:${VLLM_VERSION}
+    container_name: vllm-rerank-bge
+    restart: unless-stopped
+    ipc: host
+    ports:
+      - "${RERANK_PORT}:8000"
+    volumes:
+      - /opt/aimodels/huggingface:/hfcache
+    environment:
+      - HF_HOME=/hfcache
+      - HF_HUB_CACHE=/hfcache/hub
+      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
+      - VLLM_API_KEY=${API_KEY:-}
+    command:
+      - ${RERANK_MODEL}
+      - --served-model-name
+      - ${RERANK_MODEL}
+      - --runner
+      - pooling
+      # bge-reranker-v2-m3 is natively a cross-encoder; no --hf-overrides.
+      - --host
+      - 0.0.0.0
+      - --port
+      - "8000"
+      - --gpu-memory-utilization
+      - ${RERANK_GPU_MEM_UTIL}
+      - --max-model-len
+      - ${RERANK_MAX_MODEL_LEN}
+      - --dtype
+      - auto
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              device_ids: ["0"]
+              capabilities: [gpu]
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 180s
+    labels:
+      - homepage.group=AI - Eval & Retrieval
+      - homepage.name=vLLM Rerank bge-v2-m3 (esh-ml1)
+      - homepage.icon=mdi-sort-variant
+      - homepage.description=bge-reranker-v2-m3 via vLLM (esh-ml1, RTX 2000E Ada)
+      - homepage.href=http://10.0.50.80:${RERANK_PORT}/docs

stacks/litellm/conf/config.yaml

@@ -443,11 +443,31 @@ model_list:
   #     explicitly. Operator ruling 2026-08-23. ---
 
   # --- Qwen3 embeddings ---
+  # TWO deployments, one name, ordered: fv-ml1 serves (order 1); esh-ml1 takes
+  # over only when fv-ml1 fails (order 2 — LiteLLM's order-based fallback, v1.97
+  # router.py). Same model, same vLLM version and flags at both sites, so the
+  # vectors are interchangeable: measured 2026-09-24, cosine FV-vs-ESH median
+  # 0.999908 (min 0.999772) over 11 texts, inside the FV-vs-FV self-noise floor
+  # (median 0.999927, min 0.999791); different-text negative control 0.07–0.38.
+  # ⚠ Failover cost, measured n=3 each: primary REFUSING → +0.15 s; primary
+  # HOST DOWN (no ARP) → ~18.7 s on EVERY call (no cooldown kicked in). Slow,
+  # not broken. ⚠ Never add a deployment here that serves a DIFFERENT embedding
+  # model — vectors are model-specific and a mixed group corrupts indexes.
+  # esh-ml1: stacks/embed-rerank, playbooks/esh-ml1-lxc.yaml.
   - model_name: qwen3-embedding
     litellm_params:
       model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
       api_base: http://10.251.50.54:8001/v1
       api_key: os.environ/VLLM_API_KEY
+      order: 1
+    model_info:
+      mode: embedding
+  - model_name: qwen3-embedding
+    litellm_params:
+      model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
+      api_base: http://10.0.50.80:8001/v1
+      api_key: os.environ/VLLM_API_KEY
+      order: 2
     model_info:
       mode: embedding
 
@@ -487,12 +507,24 @@ model_list:
           enable_thinking: false
     model_info:
       mode: chat
-  # reranker → generic capability name for rerank (currently qwen3-reranker).
+  # reranker → generic capability name for rerank (bge-reranker-v2-m3 since the
+  # R43 cutover). fv-ml1 order 1, esh-ml1 order 2 — same failover shape as
+  # qwen3-embedding above. Parity measured 2026-09-24: |score| FV-vs-ESH max
+  # 0.000145 vs FV-vs-FV floor 0.000181, identical ranking over 11 documents.
   - model_name: reranker
     litellm_params:
       model: hosted_vllm/BAAI/bge-reranker-v2-m3
       api_base: http://10.251.50.54:8013/v1
       api_key: os.environ/VLLM_API_KEY
+      order: 1
+    model_info:
+      mode: rerank
+  - model_name: reranker
+    litellm_params:
+      model: hosted_vllm/BAAI/bge-reranker-v2-m3
+      api_base: http://10.0.50.80:8013/v1
+      api_key: os.environ/VLLM_API_KEY
+      order: 2
     model_info:
       mode: rerank