feat(nh3-ml1): second TEI embed/rerank backend live on nh3-pve; parity-verified vs esh-ml1

NH3 site visit done: Secure Boot off, iGPU restored as boot VGA, AMT port cabled. - nh3-pve: NVIDIA 580.178.04 (DKMS, open modules) via pve-nvidia-host.yaml. - nh3-ml1 = CT 109 @ 10.100.50.80 via gpu-lxc.yaml; embed-rerank (TEI 1.9.4) deployed with HOST_NAME/HOST_IP labels. - Parity vs esh-ml1 (1,126 texts, 2 runs/host, controls): embed cosine min 0.999993 = own noise floor; overlap@10 1.000 vs MRL-256 positive control 0.684; rerank top-1 1.00, max diff 0.0014 vs floor 0.0020. On-box speed identical within rep spread. - gpu-lxc.yaml: first step upgrades lxc-pve to >= 6.0.0-2 (Proxmox fix #7006). With 6.0.0-1 every docker run in a nesting CT failed on runc 1.5's sysctl reopen; applied on nh3-pve (one package). - pve-nvidia-host.yaml: document that the headers meta drags in the newest kernel (nh3-pve went 6.8.12-11 -> -43 at the next reboot). - Monitoring: Beszel NVIDIA agent + 5 alerts, Kuma #29/#30, Homepage nh3-ml1-docker, Dozzle agent (hub 8 clients). DNS nh3-ml1.nh3.internal. - nh3-pve README: SB/IGFX/driver/kernel state, btmtk oops on -4x kernels, AMT cabled but unreachable on the network. Gateway routing to nh3-ml1 is not changed.
2026-09-25 15:55:33 -07:00
parent 6fa8213c20
commit 5960526c3f
17 changed files with 815 additions and 112 deletions
@@ -367,6 +367,7 @@ Observed and standardized across servers:
 | esh-pve-nas | 10.0.50.55 | ESH home lab | Proxmox VE hypervisor (storage / media) | `servers/esh-pve-nas/README.md` |
 | esh-vm-db | 10.0.50.60 | ESH home lab | DB VM — PostgreSQL (paperless-ng) + MongoDB; bare-metal VM, no Docker | `servers/esh-vm-db/README.md` |
 | esh-ml1 | 10.0.50.80 | ESH home lab | GPU LXC (CT 110 on esh-pve, RTX 2000E Ada 16 GB, host-side NVIDIA driver) — **the fleet's embedding + reranking service** (TEI), sole gateway backend since 2026-09-25 | `servers/esh-ml1/README.md` |
+| nh3-ml1 | 10.100.50.80 | NH3 | GPU LXC (CT 109 on nh3-pve, RTX 2000E Ada 16 GB, host-side NVIDIA driver) — second embed/rerank (TEI) backend, twin of esh-ml1; not yet behind the gateway (2026-09-25) | `servers/nh3-ml1/README.md` |

 **Placement rules:**
 - GPU-required stacks → `fv-ml1` (primary, Fountain Valley) or `irv-ml1`
@@ -99,7 +99,7 @@ hosts:
  - {name: esh-pve,          site: esh, v4: 10.0.250.35,    note: Proxmox hypervisor}
  - {name: esh-pve-nas,      site: esh, v4: 10.0.50.55,     note: Proxmox hypervisor — storage/media}
  - {name: esh-vm-db,        site: esh, v4: 10.0.50.60,     note: PostgreSQL + MongoDB}
-  - {name: esh-ml1,          site: esh, v4: 10.0.50.80,     note: CT110 on esh-pve — RTX 2000E Ada, embedding + reranking (vLLM)}
+  - {name: esh-ml1,          site: esh, v4: 10.0.50.80,     note: CT110 on esh-pve — RTX 2000E Ada, embedding + reranking (TEI) + reward seat}
  - {name: vm-esh-nas,       site: esh, v4: 10.0.50.154,    note: NAS-adjacent docker host}
  - {name: esh-filebot,      site: esh, v4: 10.0.50.70,     note: restic / file-sync VM}
  - {name: esh-gw,           site: esh, v4: 10.0.250.1,     note: esh-gw}
@@ -111,6 +111,7 @@ hosts:
  - {name: ana-scale,        site: ana, v4: 10.250.50.45,   note: CT114 on pfi-pve — headscale/tailscale subnet router for 10.250.0.0/16}
  - {name: nh3-headscale,    site: nh3, v4: 10.100.50.45,   note: CT106 on nh3-pve — headscale control plane (public headscale.phasefinal.com:443)}
  - {name: nh3-scale,        site: nh3, v4: 10.100.50.46,   note: CT107 on nh3-pve — subnet router for 10.100.0.0/16}
+  - {name: nh3-ml1,          site: nh3, v4: 10.100.50.80,   note: CT109 on nh3-pve — RTX 2000E Ada, second embedding + reranking backend (TEI)}
  - {name: esh-scale,        site: esh, v4: 10.0.50.65,     note: CT108 on esh-pve — subnet router for 10.0.0.0/16}

 # Service aliases — a name that points at whatever host currently runs it, so
@@ -0,0 +1,89 @@
+# `[2026-09-25]` nh3-ml1 LIVE after the NH3 site visit
+
+**Prime, ~1524 PT:** "secure boot disabled, amt enabled, card plugged." This is the
+resume of `2026-09-25-nh3-ml1-standup.md`. nh3-pve came up at 1523.
+
+## Post-boot checklist: all green
+
+- Uplink is `enp2s0f1np1`. Guests 100, 101, 102, 105 and CTs 103, 106, 107 are up;
+  104 and 108 are off as expected.
+- NFS automounts are good on nh3-docker, nh3-dev and pbs-nh3. The post office is
+  push/reachable. The 14 queued althing messages were all Beszel/Kuma "UP"
+  recoveries, so nothing needed acting on.
+- Secure Boot is disabled. `00:02.0` is back with `boot_vga=1` and i915 loaded.
+
+## Build (elway, both playbooks idempotent)
+
+1. `pve-nvidia-host.yaml`: the driver installed (DKMS, 36 s), 6/6 verify.
+2. `gpu-lxc.yaml`: CT 109 built, then **verify 3 FAILED**. `docker run` died with
+   `open sysctl net.ipv4.ip_unprivileged_port_start file: reopen fd 8: permission
+   denied`.
+   - The cause is the lxc-pve version, not runc. Both CTs run runc 1.5.1. nh3-pve
+     (PVE 8.4.1) had lxc-pve **6.0.0-1**, and esh-pve (8.4.20) has **6.0.0-2**,
+     whose changelog is Proxmox "fix #7006: do not restrict /proc or /sys if
+     nesting is enabled".
+   - **Decided autonomously:** upgrade that one package (a dry-run showed
+     1 upgraded, 0 new, 0 removed) and `pct reboot 109`. GPU-in-docker then passed.
+     ops-log recorded.
+   - The playbook gained a first step that upgrades lxc-pve when it is below
+     6.0.0-2, with a quoted name, because an unquoted `#7006` truncated it as a YAML
+     comment. The re-run was 5/5 verify OK. The guard skips on esh-pve (tested).
+3. `embed-rerank` deploy. `deploy-stack.sh --yes` was **denied by the auto-mode
+   classifier as a blind apply**, so the pattern now is: preview with
+   `echo n | deploy-stack.sh …` (it shows the diff and aborts), then `echo y |`.
+   `.env` = example + `HOST_NAME=nh3-ml1`, `HOST_IP=10.100.50.80`. Healthy in about
+   120 s including the image pull; 2.7 GB VRAM.
+
+## Parity and speed (vs esh-ml1)
+
+Script: session scratchpad, not committed. Corpus: 1,120 repo-doc paragraphs
+plus 6 fixed texts, 2 runs per host, 50 queries.
+- Embedding cosine esh↔nh3: median 0.999998, min 0.999993. The floors are esh
+  0.999995 and nh3 0.999993.
+- Negative control (different texts): median 0.50.
+- overlap@10 = 1.000 both ways. Positive control, MRL-256: 0.684.
+- Rerank, 100×20: top-1 1.00, max score diff 0.0014 (the esh-esh floor is
+  0.0020). Positive control, a 4-word query: top-5 order 0.09, max diff 0.97.
+- Speed on-box, 3 interleaved reps: short embed p50 6.6–6.7 vs 7.0 ms; bulk
+  104–105 vs 106–108 passages/s; rerank-20 p50 156–159 vs 152–161 ms. The same.
+
+## Monitoring and wiring
+
+- Beszel: agent (`hosts/nh3-ml1.yaml`, same hub key as esh-ml1), system
+  `1feeeq61g4mkqre`, 5 alerts, GPU samples present.
+  - ⚠ zsh gotcha: `for a in "X 1 2"; do set -- $a` does NOT word-split in zsh, so
+    the first batch of alert POSTs failed silently (empty fields). Use a function
+    with explicit args.
+- Kuma #29/#30 (the seed also refreshed the esh-ml1 descriptions). Homepage
+  `nh3-ml1-docker` shows 2 cards with no restart. Dozzle agent added, and the hub
+  `.env` backed up as `.env.bak-20260925-nh3ml1` before the edit.
+- DNS `nh3-ml1.nh3.internal` on all 3 resolvers. `~/.ssh/config` `Host nh3-ml1`;
+  `servers/nh3-ml1/` has ssh-target, system-details and README.
+
+## Found along the way
+
+- **Kernel 6.8.12-43 came from our own playbook.** The headers step's
+  `proxmox-headers-6.8` install upgraded the `proxmox-kernel-6.8` meta
+  (apt history 14:19:23). The visit reboot booted it. Documented in
+  `pve-nvidia-host.yaml`.
+- **btmtk Bluetooth oops on every -4x boot** (2/2 lines on each of 3 -43 boots,
+  0 on the -11 boots; esh-pve -42 is the same). It taints `D`. Only the BT worker
+  dies. Proposed fix: blacklist btusb on both PVE hosts. Not done.
+- **AMT is not reachable.**
+  - The LM port has a 1 Gb carrier (brief unbridged `ip link up` with IPv6
+    disabled, then restored; ops-log recorded).
+  - No UDM client or lease for `58:47:ca:76:96:0e`.
+  - A /dev/tcp sweep of 16992/16993 over 10.100.{0,10,50,250}.0/24 found nothing.
+    Its positive controls, :8390 and :8006, did answer.
+  - The USW Pro 24's up ports (19, 22, 23, 26) are all other devices. So the cable
+    is probably on nh3-sw1 (`10.100.250.2`), where SSH times out and no creds are
+    held.
+  - Asked Prime.
+
+## Open (Prime's)
+
+- Gateway routing for `qwen3-embedding` / `reranker`. Recommendation: load-share
+  (identical cards, and a second site).
+- AMT network settings and switch port. The NanoKVM stays on nh3-pve until AMT KVM
+  shows the console.
+- btusb blacklist on nh3-pve and esh-pve. Low priority.
@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management

-_Last updated: 2026-09-25 ~1430 PT (⭐⭐ nh3-pve SHUT DOWN by Prime for the NH3 site visit — post-boot checklist in Current state. nh3-ml1 blocked on nh3-pve SECURE BOOT; Prime's SB call pending. NanoKVM blind until IGFX fix on the visit; gx10 AC-restore VALIDATED; GPU playbooks host-generic.)_
+_Last updated: 2026-09-25 ~1555 PT (⭐⭐ NH3 visit DONE: SB off, IGFX restored, AMT cabled but NOT on the network. nh3-ml1 LIVE + parity-verified + monitored; gateway routing = Prime's call. lxc-pve fix; kernel -43 btmtk oops noted. gx10 AC-restore VALIDATED.)_

 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -115,57 +115,44 @@ no longer deployed sidecars here. See Recent decisions.)

 ## Current state / in-flight

-_As of 2026-09-25 ~1430 PT._
+_As of 2026-09-25 ~1555 PT._

-### ⭐⭐ nh3-pve SHUT DOWN 2026-09-25 ~1430 PT (Prime's order) for the NH3 site visit
+### ⭐⭐ NH3 site visit DONE; nh3-pve back up 1523; nh3-ml1 LIVE (not yet in the gateway)

-This session runs ON nh3-dev and dies with it. nh3-dev wipes /tmp at boot, so there
-is NO /tmp handoff after this. **Successor: work this list first.**
-
-**The site-visit plan.** Prime has mini-DP→DP adapters, so a DisplayPort monitor on
-the RTX plus a USB keyboard covers all of it:
-1. BIOS: set Primary Display = IGFX. The Auto setting hid the iGPU, so the
-   NanoKVM is blind.
-2. BIOS/Secure Boot: Prime's call is PENDING. (1) SB off (recommended; matches
-   esh-pve), (2) keep SB and enroll a DKMS MOK via MokManager, or (3) a VFIO VM
-   instead of the LXC.
-3. vPro: cable I226-LM `enp88s0`, provision MEBx, confirm AMT KVM shows the console.
-4. Only then move the NanoKVM to pfi-gx10.
-Detail and order: `servers/nh3-pve/README.md`.
-
-**Post-boot checklist:**
-1. Network: `ping 10.100.250.60`; `ip -br link`. ⚠ The uplink is now
-   `enp2s0f1np1` (the cable moved cages on 09-25 at 1116), not `enp2s0f0np0`. Both
-   are in vmbr0.
-2. GPU: `lspci` shows `01:00.0` RTX 2000E. If IGFX was set, `00:02.0` is back with
-   `boot_vga=1`. `mokutil --sb-state` shows whether Secure Boot is off.
-3. Guests running: 100 nh3-docker1, 101 nh3-extdev, 102 nh3-dev, 105 pbs-nh3,
-   CT 103 nh3-wg, 106 nh3-headscale, 107 nh3-scale. Staying off: 104, 108.
-4. NFS automounts: nh3-docker `/mnt/compose` `/mnt/backup`; nh3-dev `/mnt/backup`
-   `/mnt/smithy`; pbs-nh3 datastore.
-5. Post office `postbox status` shows push/reachable; Kuma is green.
-6. nh3-dev services: svos :8770, hermes-gateway (Miranda's channel), Booth :8090,
-   talk, ttyd RO/RW, zellij-web, `dev-backup.timer`.
-7. Claude sessions: relaunching them is Prime's call.
-8. pfi-gx10: the AC-restore patch is validated, so a power cut brings it back on
-   its own.
-9. Resume nh3-ml1, only once Secure Boot is resolved.
-   `scripts/elway root@nh3-pve.nh3.internal --playbook playbooks/pve-nvidia-host.yaml`.
-   Its pre-flight refuses while SB is on without an enrolled key. Then run
-   `playbooks/gpu-lxc.yaml` with the nh3-ml1 `--var` line from its header, deploy
-   `embed-rerank` with `.env` `HOST_NAME=nh3-ml1` `HOST_IP=10.100.50.80`, run a
-   parity check against esh-ml1 (noise floor plus controls), then wire monitoring,
-   DNS and `servers/nh3-ml1/README.md`. Gateway routing is Prime's call; the
-   recommendation is load-share after parity.
-
-**Left on nh3-pve by the failed 09-25 driver install** (harmless, and needed later):
-headers 6.8.12-11 plus the series meta, dkms, build-essential; nouveau
-blacklisted and unloaded; the `.run` staged in `/root/nvidia`.
+Prime, after the visit: "secure boot disabled, amt enabled, card plugged." The
+post-boot checklist is all green: the uplink is on `enp2s0f1np1`, all 7 onboot
+guests are up, NFS automounts, the post office is push/reachable, and svos,
+hermes-gateway, Booth, ttyd, zellij-web and dev-backup are running.
+- **Secure Boot OFF. IGFX restored** (`00:02.0` `boot_vga=1`), so the NanoKVM should
+  have video again (not checked). NVIDIA 580.178.04 DKMS is on the host.
+- **nh3-ml1 = CT 109 @ 10.100.50.80, TEI embed+rerank, parity-verified vs esh-ml1:
+  the two hosts cannot be told apart and run at the same speed.** Monitoring is
+  wired: Beszel `1feeeq61g4mkqre` with 5 alerts, Kuma #29/#30, Homepage cards,
+  Dozzle agent (hub 7→8 clients). DNS `nh3-ml1.nh3.internal` is set.
+  `servers/nh3-ml1/README.md` exists.
+- ⏳ **Prime's call: gateway routing.** Recommendation: load-share
+  `qwen3-embedding` + `reranker` across esh-ml1 and nh3-ml1.
+- ⚠ **AMT is cabled (1 Gb link) and reported enabled, but not on the network.** No
+  UDM lease for `…:96:0e`, and no 16992/16993 on the four NH3 /24s (the sweep was
+  positive-controlled). It is not on the USW Pro 24, so probably nh3-sw1. Asked
+  Prime for the MEBx network settings and which switch port. **The NanoKVM stays
+  on nh3-pve until AMT KVM is confirmed.**
+- Found and fixed: **lxc-pve 6.0.0-1 broke Docker in the CT** (runc 1.5 sysctl
+  reopen denied). Upgraded that one package to 6.0.0-2 (Proxmox fix #7006), and
+  `gpu-lxc.yaml` now does this itself.
+- Found, NOT fixed: the headers install on 09-25 pulled kernel **6.8.12-43**, which
+  booted at the visit. It oopses in Bluetooth (`btmtk`) on every boot, and so does
+  esh-pve on -42. Benign so far. The proposed fix is to blacklist `btusb` on both
+  PVE hosts, effective at the next boot.
+- Not yet proven: nh3-ml1 coming back by itself after an nh3-pve reboot.
+  `nvidia-persistenced` is ordered before `pve-guests`, but only the config was
+  checked; nothing has been rebooted.
+→ `persistent-memory.d/2026-09-25-nh3-ml1-live.md`

 ### ⭐ esh-ml1 = the fleet's utility GPU host (2026-09-25): embed/rerank (TEI) + reward (vLLM)

-TEI is the fleet embed/rerank engine (Prime) and esh-ml1 is the SOLE backend
-(⚠ no failover until the second RTX 2000 arrives). The Skywork reward seat moved
+TEI is the fleet embed/rerank engine (Prime) and esh-ml1 is the SOLE gateway
+backend (nh3-ml1 is live and parity-verified; routing it in is Prime's call). The Skywork reward seat moved
 there too; the audit found nothing supersedes it. fv-ml1 GPU 1 freed ~15 GB
 across the day; only `vllm-coder` is left in `stacks/vllm`, and **parakeet stays**
 on fv-ml1. Monitoring is wired: Beszel with GPU and 5 alerts, Kuma #27/#28,
@@ -198,8 +185,7 @@ in `servers/pfi-gx10/README.md`. → `persistent-memory.d/2026-09-24-gx10-ac-res

 ### Live threads

- **Claude sessions on nh3-dev:** Prime relaunched 3 after the 1116 boot. They die again
-  with the ~1430 shutdown, and relaunching them is his call.
+- **Claude sessions on nh3-dev:** the ~1430 shutdown killed them again; relaunching is Prime's call.
 - **Homepage `hermes-gateway seat` rename resolved (2026-09-25):** the operator's 2026-09-24 handle split is real (`postbox handles` records it; infra-hermes confirmed the seat daemon carries `hermes-gateway`). The rename was already live and is committed as `422a27c`.
 - **High Seat backup proof DONE 2026-09-25 0111:** infra-hermes reported and I verified that restic snapshot `cd113bef` (nh3-dev, 0100) contains `~/.high-seat/reports/grok.963ddd3f428b7fc7.jsonl`. Told him to retire the daily 0110 check unless svos-dev asked for an ongoing one.
 - ESH has a single outside route (esh-scale on esh-pve) — noted, untracked.
@@ -207,6 +193,7 @@ in `servers/pfi-gx10/README.md`. → `persistent-memory.d/2026-09-24-gx10-ac-res

 ## Recent decisions

+- `[2026-09-25]` **nh3-ml1 LIVE after the NH3 visit (SB off, IGFX restored): driver + CT 109 + TEI, parity vs esh-ml1 indistinguishable (cos min 0.999993 = own floor; overlap@10 1.000 vs MRL-256 control 0.684), same speed; monitoring/DNS wired. Autonomous: lxc-pve 6.0.0-2 upgrade on nh3-pve (Docker-in-LXC fix #7006). Gateway routing + AMT follow-up are Prime's.** → `persistent-memory.d/2026-09-25-nh3-ml1-live.md`
 - `[2026-09-25]` **nh3-ml1 = CT 109 @ 10.100.50.80, second TEI embed/rerank backend on nh3-pve's RTX 2000E; GPU playbooks made host-generic — BUILD DEFERRED: blocked on nh3-pve Secure Boot until the NH3 visit; SB handling and gateway routing are Prime's calls.** Tracked: `7ddd116`, Current state. → `persistent-memory.d/2026-09-25-nh3-ml1-standup.md`
 - `[2026-09-25]` **nh3-pve OOB: HOLD blind until the next NH3 visit (Prime via Miranda, 1333).** Target: NanoKVM → pfi-gx10, and the MS-01 on its own vPro. The IGFX BIOS fix rides the same visit, since AMT KVM captures only the iGPU. Tracked: `servers/nh3-pve/README.md` (`2b49be4`, `cdd7605`).
 - `[2026-09-25]` **MS-01 foot-gun: a GPU in the PCIe slot renames every NIC** (the slot's root port takes bus 01, so the X710 goes `enp2s0f0np0`→`enp3s0f0np0`). On nh3-pve that means booting with no uplink. Pinned by MAC with `.link` files before the install, and the AMT port was pulled from vmbr0. → `servers/nh3-pve/README.md`, `playbooks/nh3-pve-pin-nic-names.yaml` (`2118449`)
@@ -65,6 +65,25 @@ vars:
  infra_ops_pubkey: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN+1HBwfXrkfTYWdcnWCjLJ6VLAGC87gxH5h5vKaaA3c infra-ops@pfi-fleet"

 steps:
+  # runc >= 1.2.8 (the CVE-2025-52881 fix; the CT gets 1.5.x from docker-ce)
+  # re-opens /proc/sys files, and lxc-pve 6.0.0-1's AppArmor profile denies it
+  # in a nesting CT: EVERY `docker run` fails with "open sysctl
+  # net.ipv4.ip_unprivileged_port_start file: reopen fd 8: permission denied".
+  # lxc-pve 6.0.0-2 (Proxmox fix #7006) lifts the restriction when nesting is on.
+  # Measured 2026-09-25: nh3-pve (PVE 8.4.1, lxc-pve 6.0.0-1) failed the GPU
+  # verify below; esh-pve (8.4.20, 6.0.0-2) never did. One package, no deps,
+  # running guests unaffected (the change only relaxes the profile). A CT that
+  # was already running needs a `pct reboot` to pick it up.
+  - name: "lxc-pve carries fix #7006 (Docker's runc works in a nesting CT)"
+    shell: |
+      set -e
+      export DEBIAN_FRONTEND=noninteractive
+      apt-get update -qq
+      apt-get install -y -qq --only-upgrade lxc-pve
+      v=$(dpkg-query -W -f='${Version}' lxc-pve)
+      dpkg --compare-versions "$v" ge 6.0.0-2 || { echo "lxc-pve is still $v after upgrade" >&2; exit 1; }
+    when: "dpkg --compare-versions \"$(dpkg-query -W -f='${Version}' lxc-pve)\" lt 6.0.0-2"
+
  - name: Fetch the Debian 12 template
    shell: pveam update >/dev/null && pveam download local {{ template }}
    creates: /var/lib/vz/template/cache/{{ template }}
@@ -72,6 +72,12 @@ steps:
  # DKMS needs the headers for the RUNNING kernel (built now) and the series
  # meta-package (so each future kernel's headers arrive with it and DKMS
  # rebuilds on upgrade). nh3-pve had neither, nor dkms or a compiler.
+  # ⚠ The series meta depends on the NEWEST headers, which drags the
+  # proxmox-kernel meta and the newest kernel image in with it. On nh3-pve
+  # (2026-09-25) this installed 6.8.12-43 next to the running -11, and the next
+  # reboot silently booted -43. Expect a kernel change on the next reboot of any
+  # host that was behind, and before that reboot check that `dkms status` lists
+  # nvidia for the kernel that will boot, not only for `uname -r`.
  - name: Kernel headers (running kernel + series meta), dkms, build tools
    shell: |
      set -e
@@ -5,10 +5,12 @@ GPU LXC for the ESH home lab: **CT 110 on esh-pve**, holding the **NVIDIA RTX
 reranking service** — the only backend behind the gateway's `qwen3-embedding`,
 `reranker` and `reranker-a3-bge-v2-m3` since 2026-09-25. Built 2026-09-24.

-⚠ **Single backend until the second RTX 2000 arrives** (Prime, 2026-09-25). If
-esh-ml1, esh-pve, or ESH's mesh route (esh-scale, CT 108) is down, fleet
-embeddings and reranking are down: Worldtree recall, nevermore clustering, Open
-WebUI RAG.
+⚠ **Still the single gateway backend.** The second RTX 2000 is live as
+[nh3-ml1](../nh3-ml1/README.md) (2026-09-25, parity-verified: the two hosts cannot
+be told apart), but LiteLLM does not route to it yet; that is Prime's call. Until
+it does, if esh-ml1, esh-pve, or ESH's mesh route (esh-scale, CT 108) is down,
+fleet embeddings and reranking are down: Worldtree recall, nevermore clustering,
+Open WebUI RAG.

 | | |
 |---|---|
@@ -0,0 +1,105 @@
+# nh3-ml1
+
+GPU LXC for the NH3 site: **CT 109 on nh3-pve**, holding the **NVIDIA RTX 2000E
+Ada** (16 GB, 50 W, `01:00.0`, PCIe gen4 x8). **It is the second embedding and
+reranking backend**, the twin of [esh-ml1](../esh-ml1/README.md): same card, same
+driver, same TEI image and models. Built 2026-09-25, after the NH3 site visit
+turned Secure Boot off on nh3-pve.
+
+⚠ **Not behind the gateway yet.** It serves on its own ports and is monitored, but
+LiteLLM still routes `qwen3-embedding` and `reranker` to esh-ml1 alone. Routing is
+Prime's call (recommendation: load-share; see below).
+
+| | |
+|---|---|
+| **IP** | `10.100.50.80/24`, VLAN 50 (`nh3-servers`), gateway `10.100.50.1` (static, outside the UDM's `.150–.249` DHCP pool) |
+| **DNS** | `nh3-ml1.nh3.internal` |
+| **SSH** | `ssh nh3-ml1` → `infra-ops@10.100.50.80` (NOPASSWD sudo) · from the host: `pct enter 109` |
+| **OS** | Debian 12, unprivileged, `nesting=1,keyctl=1` |
+| **Size** | 6 cores, 16 GB RAM + 2 GB swap, 80 GB rootfs on `local-zfs` |
+| **Boot** | `onboot: 1`, `startup: order=30`, after the site's core guests, so a GPU fault never delays NH3's DNS or mesh route |
+| **Backups** | **None, on purpose.** nh3-pve's vzdump job is `all 1`; the playbook added 109 to its `exclude` list. Everything is rebuilt from the playbooks and the stack; models re-download. |
+
+## What it serves
+
+`stacks/embed-rerank` (`/opt/docker/compose/embed-rerank`), **TEI 1.9.4**
+(`89-1.9.4`). The live `.env` differs from `.env.example` only in
+`HOST_NAME=nh3-ml1` and `HOST_IP=10.100.50.80`, which feed the Homepage labels.
+
+| container | model | port |
+|---|---|---|
+| `tei-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 (`/v1/embeddings`, `/embed`) |
+| `tei-rerank` | `BAAI/bge-reranker-v2-m3` | 8013 (`/rerank`, body `query` + `texts`) |
+
+VRAM ~2.7 GB for both, so ~13 GB is free.
+
+## Parity and speed vs esh-ml1 (2026-09-25 ~1540 PT)
+
+Client on nh3-dev, calling both seats directly. Corpus: 1,120 paragraphs from
+this repo's docs plus 6 fixed texts (CJK, code, emoji, a 1-char input, a 6k-char
+passage); 50 instruction-format queries; each host embedded everything twice.
+
+| embedding check | result | noise floor / control |
+|---|---|---|
+| per-text cosine, esh vs nh3 | median 0.999998, min 0.999993 | esh vs esh 0.999998 / 0.999995; nh3 vs nh3 0.999998 / 0.999993 |
+| overlap@10, nh3 queries on the esh index | 1.000 | esh rerun 1.000; **positive control** MRL-256 truncation 0.684 |
+| overlap@10, nh3 index + nh3 queries | 1.000 | — |
+| hit@1 own paragraph | 0.76 (both hosts) | — |
+| **negative control**, different texts | cosine median 0.50 | — |
+
+| rerank check (100 queries × 20 docs) | esh vs nh3 | esh vs esh | **positive control** (query cut to 4 words) |
+|---|---|---|---|
+| top-1 agreement | 1.00 | 1.00 | 0.94 |
+| top-5 exact order | 0.98 | 0.97 | 0.09 |
+| max score difference | 0.0014 | 0.0020 | 0.97 |
+
+**The two hosts cannot be told apart.** Every esh-vs-nh3 figure sits inside the
+esh-vs-esh noise. Sensitivity floor: this method cannot resolve an embedding
+cosine gap below ~5×10⁻⁶ or a rerank score difference below ~0.002. An index
+built on either host serves queries from the other.
+
+**Speed, on-box**, 3 interleaved reps per host (range across reps):
+
+| workload | nh3-ml1 | esh-ml1 |
+|---|---|---|
+| embed 1 short query, p50 | 6.60–6.74 ms | 6.97–7.02 ms |
+| bulk embed, 64 per request, passages/s | 103.7–105.0 | 105.5–108.1 |
+| rerank 20 docs, p50 | 155.8–159.2 ms | 151.8–160.5 ms |
+
+The two hosts run at the same speed. esh-ml1 was also holding the idle reward seat
+(~8 GB VRAM, 0% util) during these runs.
+
+**Gateway routing (Prime's call).** The recommendation is **load-share**. The
+2026-09-25 rule against load-sharing came from pairing esh-ml1 with the much
+faster fv-ml1. These two cards are identical, and a second site removes the
+single-host outage the esh-ml1 README warns about.
+
+## How it is built
+
+1. [`playbooks/pve-nvidia-host.yaml`](../../playbooks/pve-nvidia-host.yaml) on
+   nh3-pve: driver **580.178.04** (open modules, DKMS) plus the
+   `nvidia-persistenced` unit. It needs **Secure Boot off**, which was turned off
+   in the BIOS on the 2026-09-25 visit; the pre-flight refuses otherwise.
+2. [`playbooks/gpu-lxc.yaml`](../../playbooks/gpu-lxc.yaml) with the
+   "Run (nh3-ml1)" `--var` line from its header.
+   ⚠ nh3-pve was on **lxc-pve 6.0.0-1**. With it, every `docker run` in the CT
+   failed with *"open sysctl net.ipv4.ip_unprivileged_port_start file: reopen
+   fd 8: permission denied"* (runc 1.5 against the old AppArmor profile).
+   Upgrading lxc-pve alone to 6.0.0-2 (Proxmox fix #7006) and then running
+   `pct reboot 109` fixed it. The playbook now does the upgrade as its first step.
+3. `scripts/deploy-stack.sh nh3-ml1 embed-rerank`, then set `HOST_NAME` /
+   `HOST_IP` in `.env`, then `docker compose up -d`.
+
+The **driver version lock** and the DKMS/kernel notes in
+[esh-ml1's README](../esh-ml1/README.md#-driver-version-lock) apply here unchanged.
+nh3-pve runs kernel 6.8.12-43, which the 2026-09-25 headers install pulled in
+(see `servers/nh3-pve/README.md`).
+
+## Monitoring and telemetry (wired 2026-09-25)
+
+| layer | what | where |
+|---|---|---|
+| **Beszel** | NVIDIA agent `henrygd/beszel-agent-nvidia:0.18.7`, `stacks/beszel` + `hosts/nh3-ml1.yaml`, hub system `1feeeq61g4mkqre`; GPU util, VRAM and power are sampled | alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m |
+| **Uptime Kuma** | `Embed — Qwen3 0.6B (TEI, nh3-ml1)` → `:8001/health` (#29); `Rerank — bge-v2-m3 (TEI, nh3-ml1)` → `:8013/health` (#30) | `stacks/uptimekuma/monitors.yaml` |
+| **Homepage** | two cards under *AI - Eval & Retrieval*; dockerd on tcp/2375 bound to `10.100.50.80` | `stacks/homepage/conf/docker.yaml` → `nh3-ml1-docker` |
+| **Dozzle** | agent `v10.4.1` on `10.100.50.80:7007`, compose dir `dozzle-agent`; added to the hub's `DOZZLE_REMOTE_AGENT` | hub on ana-docker :8088 |
@@ -0,0 +1 @@
+infra-ops@10.100.50.80
@@ -13,7 +13,7 @@ Proxmox VE hypervisor for the NH3 site (`nh3-vmhost.phasefinal.com`).

 - **CPU:** 13th Gen Intel Core i9-13900H
 - **RAM:** 62.5 GB
- **Kernel:** `6.8.12-11-pve` (Proxmox 8.x)
+- **Kernel:** `6.8.12-43-pve` since 2026-09-25 (was `6.8.12-11`; see the kernel bullet below). PVE `8.4.1`, well behind esh-pve's `8.4.20` (177 packages pending)
 - **Storage:** mostly networked — `/mnt/pve/pfi-nh3-nas` (42 TB) mounted from the Synology at `10.100.50.50:/volume1/VMStorage`; ~27 TB used

 ## What it runs
@@ -33,6 +33,7 @@ after a power loss** is the column that matters in a recovery: a guest listed as
 | 103 | nh3-wg | CT | 1 | up |
 | 106 | nh3-headscale | CT | 1 | up |
 | 107 | nh3-scale | CT | 1 | up (mesh subnet router + fleet egress proxy) |
+| 109 | nh3-ml1 (`10.100.50.80`) | CT | 1 (order 30) | up. GPU LXC, second embed/rerank backend (`servers/nh3-ml1/README.md`); needs the NVIDIA module, so it is the one guest a driver fault can stop |

 **Power-loss recovery (2026-09-24 outage).** Every guest boots at once, and
 nh3-nas is the slowest to serve NFS. NFS clients now mount nh3-nas shares on
@@ -55,12 +56,19 @@ not power back on by itself.
  vmbr0 members, so either cage works. **Never drop either port from the bridge**
  without checking which one has carrier (`ip -br link`). The bridge carries the
  I226-V's MAC `…:96:0d` because it is the first port listed.
- **AMT: NOT wired.** The AMT-capable I226-LM (`enp88s0`, MAC `58:47:ca:76:96:0e`)
-  has no cable. The ME is present (`/dev/mei0`, "AMT SOL Redirection" 00:16.3), but
-  MEBx provisioning status is unknown. To wire it: cable the LM port, then at boot
-  press Ctrl+P → set the MEBx password, enable manageability, set network (static
-  or DHCP), KVM on, User Opt-in = None, activate network access. MEBx can be driven
-  remotely through the NanoKVM below.
+- **AMT: cabled and enabled, but NOT reachable (2026-09-25 visit).** The I226-LM
+  (`enp88s0`, MAC `58:47:ca:76:96:0e`) now has a 1 Gb link (measured at 1552 by
+  bringing the port up unbridged for a few seconds). Prime reports AMT enabled in
+  MEBx. **Nothing answers on the network, though.** The UDM has no client or lease
+  for `…:96:0e`, and no host on `10.100.{0,10,50,250}.0/24` has 16992 or 16993
+  open. The sweep was checked against known-open ports and does see them. The
+  cable is not on the USW Pro 24, whose up ports are 19, 22, 23 and 26, all
+  accounted for. So it is on another switch, most likely nh3-sw1
+  (`10.100.250.2`, no infra-ops access). Likely causes: MEBx "Activate Network
+  Access" was not done, a static IP outside those subnets, or a switch port on a
+  VLAN that gets no DHCP. MEBx menu: Ctrl+P at boot → password, manageability on,
+  network (static or DHCP), KVM on, User Opt-in = None, activate network access.
+  **Keep the NanoKVM here until AMT KVM is confirmed.**
 - **Console OOB exists: a Sipeed NanoKVM** is attached (USB `3346:1009` on the host;
  web UI **`https://10.100.250.171`**, switch port 23, nh3-mgmt). It gives video and
  keyboard, so BIOS, MEBx and a host that booted without network are all reachable
@@ -80,48 +88,42 @@ not power back on by itself.
 - **`enp88s0` (the AMT port) is no longer a vmbr0 bridge port** (file edited
  2026-09-25, effective next boot). STP is off, so bridging a second cabled uplink
  into the same L2 would loop the site LAN.
- **GPU installed 2026-09-25: RTX 2000E Ada at `01:00.0`** (`10de:28b0`). The pins
-  held: the X710 moved to bus 03 and every NIC kept its name. No NVIDIA driver yet,
-  so `nouveau` binds it (`gsp ctor failed: -2` is expected without GSP firmware).
-  ⚠ **After the install the iGPU is gone from the PCI bus**: `00:02.0` enumerated on
-  the 2026-09-24 boot and is absent now, `/dev/dri` does not exist, and the RTX is
-  `boot_vga=1`. The BIOS's Auto primary display picked the PCIe card and hid the
-  iGPU, so the NanoKVM's iGPU-HDMI capture has no source. Fix: BIOS → Primary
-  Display = IGFX (or enable iGPU Multi-Monitor). Reaching the BIOS now needs a
-  display on the RTX's mini-DP, for example the NanoKVM through a mini-DP→HDMI
-  adapter. Verify with `lspci | grep 00:02.0` and `boot_vga` on `00:02.0`.
- ⚠ **Secure Boot is ON here** (`mokutil --sb-state`, lockdown `integrity`;
-  enrolled MOK = the Proxmox Secure Boot CA). esh-pve, the same model, has it
-  OFF. Any DKMS-built module, the NVIDIA driver included, is refused until its
-  key is enrolled through MokManager at boot or Secure Boot is turned off in the
-  BIOS. Both need the console, and the console is blind (below). The 2026-09-25
-  NVIDIA install failed on this and rolled itself back. What it left, all
-  harmless: headers `6.8.12-11` plus the series meta, dkms and build-essential;
-  nouveau blacklisted and unloaded; the `.run` staged in `/root/nvidia`.
- **OOB decision (Prime via Miranda, 2026-09-25): HOLD. nh3-pve stays console-blind
-  until the next NH3 site visit.** No BIOS change and no reboot until then. The
-  accepted risk: a boot without network means a site trip. **Target end state:**
-  the NanoKVM moves to pfi-gx10, and this MS-01 gets out-of-band access through its
-  own vPro/AMT (cable the I226-LM `enp88s0` and provision MEBx; see the AMT bullet
-  above).
-  ⚠ **Do the BIOS fix on that same visit.** AMT's KVM redirection captures only
-  the Intel iGPU's framebuffer, so with the iGPU hidden as it is now, AMT KVM would
-  be just as blind as the NanoKVM. Order: set Primary Display = IGFX, provision
-  MEBx, confirm AMT KVM shows the console, and only then move the NanoKVM to the
-  gx10.
-  **Reaching the BIOS** (2026-09-25 facts). The NanoKVM input is HDMI, the RTX has
-  mini-DP only, and the adapters on hand are mini-DP→DP only. The paths:
-  (1) an **active** mini-DP→HDMI adapter to feed the NanoKVM, since a passive one
-  depends on DP++; (2) a DisplayPort monitor plus a USB keyboard at the rack;
-  (3) pull the card, boot on the iGPU, set IGFX explicitly (not Auto), then refit
-  the card. The NIC pins make a card-out boot safe.
-  `systemctl reboot --firmware-setup` works here (`OsIndicationsSupported` bit 0
-  set), so nobody has to catch the Del key at POST.
-  **An OS-side patch is not possible:** the AMI `Setup`/`SaSetup` variables are not
-  runtime-visible here, unlike on the gx10.
-  **Known-good reference: esh-pve.** Same MS-01, same BIOS `AHWSA.1.17`, same RTX
-  2000E, and its `00:02.0` is present with `boot_vga=1`. The target state works on
-  this hardware.
+- **GPU: RTX 2000E Ada at `01:00.0`** (`10de:28b0`), installed 2026-09-25. The
+  NIC pins held: the X710 moved to bus 03 and every NIC kept its name. **NVIDIA
+  580.178.04** (open modules, DKMS) has been on the host since 2026-09-25 at 1527
+  (`playbooks/pve-nvidia-host.yaml`), with `nvidia-persistenced` ordered before
+  `pve-guests`. It serves CT 109 nh3-ml1.
+- **iGPU restored (2026-09-25 visit).** With the card in, the BIOS's Auto primary
+  display had hidden the iGPU. That left the NanoKVM (iGPU HDMI) blind and would
+  have blinded AMT KVM too, since AMT captures only the iGPU. It was set on the
+  visit, and since the 1523 boot `00:02.0` is back with `boot_vga=1` and i915
+  loaded. The NanoKVM should have video again (not checked from here).
+  `systemctl reboot --firmware-setup` works (`OsIndicationsSupported` bit 0), so
+  nobody has to catch Del at POST. The AMI `Setup` variables are not
+  runtime-visible, so there is no OS-side BIOS patch. esh-pve, the same MS-01 and
+  BIOS `AHWSA.1.17`, is the known-good reference.
+- **Secure Boot: OFF since the 2026-09-25 visit** (`mokutil --sb-state`: disabled),
+  which matches esh-pve. While it was ON (lockdown `integrity`), the DKMS NVIDIA
+  module was refused and the first install rolled itself back. The playbook's
+  pre-flight refuses if it is ever turned back on without an enrolled DKMS MOK.
+- ⚠ **Kernel jumped `6.8.12-11` → `6.8.12-43` at the visit reboot, pulled in by
+  our own playbook.** At 1419 the headers step ran `apt-get install
+  proxmox-headers-6.8`. That upgraded the `proxmox-kernel-6.8` meta and installed
+  `proxmox-kernel-6.8.12-43-pve-signed`. Nobody chose the new kernel; it booted
+  because it was the newest.
+  Side effect: **every -43 boot oopses in Bluetooth** (`btmtk_usb_hci_wmt_sync` →
+  NULL deref in `hci_power_on`, the MS-01's MediaTek BT; taint `D`). It hit on all
+  three -43 boots and on none of the -11 boots. esh-pve on `-42` shows the same
+  oops and has run fine, so it is benign so far: only the BT worker dies. The fix
+  is to blacklist `btusb` on both hypervisors. That is not done, because it only
+  takes effect at the next boot.
+- **lxc-pve 6.0.0-1 → 6.0.0-2** (2026-09-25 1533, that one package only). This is
+  Proxmox fix #7006. Without it, runc 1.5 inside a nesting CT fails every
+  `docker run`. `playbooks/gpu-lxc.yaml` now upgrades it first.
+- **OOB plan status.** Prime ruled on 2026-09-25 via Miranda to HOLD console-blind
+  until the site visit. Target: the NanoKVM moves to pfi-gx10, and this MS-01 uses
+  its own AMT. The visit did IGFX, turned SB off and cabled plus enabled AMT.
+  **Still open:** AMT is not on the network (above), so the NanoKVM stays here.

 ## Refresh state

@@ -24,6 +24,7 @@ filesystem samples verified; fleet 13/14 up with known fv-ml1 outage.
 | vm-esh-nas | beszel-agent-esh-nas | /mnt/books, /mnt/share, /mnt/music, /mnt/media |
 | nh3-dev | beszel | none since 2026-09-25 (was /mnt/backup, /mnt/smithy — see below) |
 | esh-ml1 (added 2026-09-25) | beszel | none — NVIDIA image (`hosts/esh-ml1.yaml`) for the RTX 2000E Ada |
+| nh3-ml1 (added 2026-09-25) | beszel | none — NVIDIA image (`hosts/nh3-ml1.yaml`) for the RTX 2000E Ada; hub system `1feeeq61g4mkqre`, same 5 alerts as esh-ml1 |

 ⚠ **nh3-dev's agent was DOWN from the 2026-09-24 NH3 power recovery until
 2026-09-25.** Docker could not bind `/mnt/smithy` at boot ("no such device"): since
@@ -0,0 +1,15 @@
+# nh3-ml1 (CT 109 on nh3-pve) — Beszel agent with NVIDIA GPU telemetry for the
+# RTX 2000E Ada (utilization, VRAM, temperature, power). Twin of hosts/esh-ml1.yaml:
+# Docker-in-LXC, NVIDIA container toolkit with no-cgroups=true
+# (playbooks/gpu-lxc.yaml). No extra filesystems: the root filesystem holds
+# everything, models included.
+services:
+  beszel-agent:
+    image: henrygd/beszel-agent-nvidia:0.18.7
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [utility]
@@ -8,6 +8,7 @@ Container log viewer. One UI on **ana-docker** aggregates logs from every Docker
 - **esh-docker-vm** / **vm-esh-nas** (agents) — `10.0.50.45:7007`, `10.0.50.154:7007`
 - **irv-ml1** (agent) — `10.6.110.50:7007` (mesh address)
 - **esh-ml1** (agent, added 2026-09-25) — `10.0.50.80:7007`, pinned `v10.4.1` = the hub's version; compose dir `dozzle-agent`. The host needed an (empty) `traefik-net` network because the compose declares it external.
+- **nh3-ml1** (agent, added 2026-09-25) — `10.100.50.80:7007`, same shape as esh-ml1 (`v10.4.1`, `dozzle-agent`, empty `traefik-net`). Hub `clients` went 7 → 8.
 - **nh3-docker** (agent, cross-site) — `10.100.50.40:7007`. ⚠ **Stopped by hand ~2026-04 (Exited 0) and left that way**; the hub logs a refused connection for it. Revive it or drop it from the list deliberately.

 ⚠ **The hub's agent list silently rots when a host moves.** Until 2026-09-25 it
@@ -3,7 +3,10 @@
 **The fleet's embedding + reranking service**, on **esh-ml1** (CT 110 on esh-pve,
 RTX 2000E Ada), served by **Hugging Face Text Embeddings Inference (TEI)**.
 Since 2026-09-25 it is the only backend behind the gateway names
-`qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3`.
+`qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3`. A second instance
+runs on **nh3-ml1** (CT 109 on nh3-pve, the same card), parity-verified against
+esh-ml1 on 2026-09-25 (`servers/nh3-ml1/README.md`). It is not in the gateway
+yet; that is Prime's call.

 **TEI is the fleet's embed/rerank engine** (Prime, 2026-09-25). New embedding or
 reranking seats go on TEI, not vLLM. Why, and the measurements behind it:
@@ -43,6 +46,9 @@ scripts/deploy-stack.sh esh-ml1 embed-rerank
 ssh esh-ml1 'cd /opt/docker/compose/embed-rerank && cp -n .env.example .env && docker compose config -q && docker compose up -d'
 ```

+nh3-ml1 is the same, plus `HOST_NAME=nh3-ml1` and `HOST_IP=10.100.50.80` in its
+`.env` (they only feed the Homepage labels).
+
 Host prerequisites (driver, LXC, docker, toolkit) are in
 [`servers/esh-ml1/README.md`](../../servers/esh-ml1/README.md).

@@ -37,6 +37,12 @@ esh-ml1-docker:
  host: 10.0.50.80
  port: 2375

+# nh3-ml1 — CT 109 on nh3-pve: the second embed/rerank (TEI) backend.
+# dockerd listens only on its own address (playbooks/gpu-lxc.yaml), 2026-09-25.
+nh3-ml1-docker:
+  host: 10.100.50.80
+  port: 2375
+
 # Example TLS socket (if/when a host moves off plaintext 2375):
 # ana-pfi-docker:
 #   host: 10.250.50.70
@@ -93,19 +93,28 @@ monitors:
    url: http://10.250.50.70:8200/api/v1/services

  # ---- fleet embed/rerank: the one EXCEPTION to "seats are OUT" ----
-  # Since 2026-09-25 these are the SOLE backends behind the gateway's
-  # `qwen3-embedding` and `reranker` (TEI on esh-ml1, no failover until the second
-  # RTX 2000 arrives). They are not come-and-go seats: dead = Worldtree recall,
-  # nevermore clustering and Open WebUI RAG all fail. TEI's /health runs the
-  # backend, so a loaded-but-broken model reads DOWN, not UP. The reward seat on the
-  # same box stays OUT: it has no working consumer (stacks/reward-seat/README.md).
+  # Since 2026-09-25 these back the gateway's `qwen3-embedding` and `reranker`
+  # (TEI on esh-ml1; nh3-ml1 is the second RTX 2000, parity-verified the same day,
+  # gateway routing pending Prime). They are not come-and-go seats: dead =
+  # Worldtree recall, nevermore clustering and Open WebUI RAG all fail. TEI's
+  # /health runs the backend, so a loaded-but-broken model reads DOWN, not UP. The
+  # reward seat on esh-ml1 stays OUT: it has no working consumer
+  # (stacks/reward-seat/README.md).
  - name: Embed — Qwen3 0.6B (TEI, esh-ml1)
    url: http://10.0.50.80:8001/health
-    description: sole backend for gateway `qwen3-embedding`
+    description: gateway `qwen3-embedding` backend (esh-ml1)

  - name: Rerank — bge-v2-m3 (TEI, esh-ml1)
    url: http://10.0.50.80:8013/health
-    description: sole backend for gateway `reranker`
+    description: gateway `reranker` backend (esh-ml1)
+
+  - name: Embed — Qwen3 0.6B (TEI, nh3-ml1)
+    url: http://10.100.50.80:8001/health
+    description: second `qwen3-embedding` backend (nh3-ml1)
+
+  - name: Rerank — bge-v2-m3 (TEI, nh3-ml1)
+    url: http://10.100.50.80:8013/health
+    description: second `reranker` backend (nh3-ml1)

  - name: talk
    url: https://talk.nh3.phasefinal.com:8092/