esh-pfi-infrastructure/servers/nh3-ml1/README.md

# nh3-ml1

GPU LXC for the NH3 site: **CT 109 on nh3-pve**, holding the **NVIDIA RTX 2000E
Ada** (16 GB, 50 W, `01:00.0`, PCIe gen4 x8). **It is the second embedding and
reranking backend**, the twin of [esh-ml1](../esh-ml1/README.md): same card, same
driver, same TEI image and models. Built 2026-09-25, after the NH3 site visit
turned Secure Boot off on nh3-pve.

**Load-shared with esh-ml1 since 2026-09-26 (Prime).** The gateway splits
`qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3` across both hosts at
random, one `simple-shuffle` deployment each. Measured with 40 + 40 requests: embed
12 esh / 28 nh3, rerank 22 / 18.

**Failover, measured by stopping this host's TEI** (2026-09-26 0747–0753):
- Without failover settings: **7/40 embeds returned 500**. The router's default 2
  retries re-picked at random and hit the dead backend again.
- With `router_settings.enable_weighted_failover: true`, which only affects
  multi-deployment groups:
  - rerank **80/80** OK.
  - embeds **38/40** in the first test (2 failures at outage onset) and **60/60**
    across a 34 s sustained outage.
  - Once TEI restarted, nh3 was back in rotation (15/30).
- Residual: in LiteLLM 1.97 the embedding path does not stamp the failed
  deployment, so embeds rely on the router's cooldown rather than per-request
  re-pick. Expect a couple of failed embeds when a backend first dies. n = 2
  outages.

| | |
|---|---|
| **IP** | `10.100.50.80/24`, VLAN 50 (`nh3-servers`), gateway `10.100.50.1` (static, outside the UDM's `.150–.249` DHCP pool) |
| **DNS** | `nh3-ml1.nh3.internal` |
| **SSH** | `ssh nh3-ml1` → `infra-ops@10.100.50.80` (NOPASSWD sudo) · from the host: `pct enter 109` |
| **OS** | Debian 12, unprivileged, `nesting=1,keyctl=1` |
| **Size** | 6 cores, 16 GB RAM + 2 GB swap, 80 GB rootfs on `local-zfs` |
| **Boot** | `onboot: 1`, `startup: order=30`, after the site's core guests, so a GPU fault never delays NH3's DNS or mesh route |
| **Backups** | **None, on purpose.** nh3-pve's vzdump job is `all 1`; the playbook added 109 to its `exclude` list. Everything is rebuilt from the playbooks and the stack; models re-download. |

## What it serves

`stacks/embed-rerank` (`/opt/docker/compose/embed-rerank`), **TEI 1.9.4**
(`89-1.9.4`). The live `.env` differs from `.env.example` only in
`HOST_NAME=nh3-ml1` and `HOST_IP=10.100.50.80`, which feed the Homepage labels.

| container | model | port |
|---|---|---|
| `tei-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 (`/v1/embeddings`, `/embed`) |
| `tei-rerank` | `BAAI/bge-reranker-v2-m3` | 8013 (`/rerank`, body `query` + `texts`) |

VRAM ~2.7 GB for both, so ~13 GB is free.

**Dataset-foundry utility seats (2026-09-26, brokkr; operator-approved):**
- `stacks/lfm-vl-seat`: LFM2.5-VL-3B on llama.cpp, `:8030`, gateway `lfm25-vl-3b`.
- `stacks/vibevoice-asr-seat`: VibeVoice-ASR-Streaming-1.5B on audio.cpp,
  `:8031`, direct only.

- `stacks/lfm-vl-uncensored-seat`: an abliterated LFM2.5-VL-3B running in
  parallel for brokkr's NSFW-caption A/B, `:8032`, direct only.

GPU total with TEI is ~11.3 of 16 GB (the ASR seat moved to Q8_0 on 2026-09-26). All of these are batch workloads, which
suits this card.

**Tried and removed (2026-09-25):** a copy of the code-completion seat
(Qwen2.5-Coder-1.5B). It gave the same quality but ran about 5× slower than on
fv-ml1 (a 64-token FIM completion took ~1.0 s vs ~0.2 s), so it stays on fv-ml1
(Prime). The recipe and measurements are in `stacks/coder-seat/README.md`.
**Lesson for the next candidate:** this card has about a fifth of a Blackwell's
decode speed on small generative models, so it suits embedding, reranking and
classification, not latency-sensitive generation.

## Parity and speed vs esh-ml1 (2026-09-25 ~1540 PT)

Client on nh3-dev, calling both seats directly. Corpus: 1,120 paragraphs from
this repo's docs plus 6 fixed texts (CJK, code, emoji, a 1-char input, a 6k-char
passage); 50 instruction-format queries; each host embedded everything twice.

| embedding check | result | noise floor / control |
|---|---|---|
| per-text cosine, esh vs nh3 | median 0.999998, min 0.999993 | esh vs esh 0.999998 / 0.999995; nh3 vs nh3 0.999998 / 0.999993 |
| overlap@10, nh3 queries on the esh index | 1.000 | esh rerun 1.000; **positive control** MRL-256 truncation 0.684 |
| overlap@10, nh3 index + nh3 queries | 1.000 | — |
| hit@1 own paragraph | 0.76 (both hosts) | — |
| **negative control**, different texts | cosine median 0.50 | — |

| rerank check (100 queries × 20 docs) | esh vs nh3 | esh vs esh | **positive control** (query cut to 4 words) |
|---|---|---|---|
| top-1 agreement | 1.00 | 1.00 | 0.94 |
| top-5 exact order | 0.98 | 0.97 | 0.09 |
| max score difference | 0.0014 | 0.0020 | 0.97 |

**The two hosts cannot be told apart.** Every esh-vs-nh3 figure sits inside the
esh-vs-esh noise. Sensitivity floor: this method cannot resolve an embedding
cosine gap below ~5×10⁻⁶ or a rerank score difference below ~0.002. An index
built on either host serves queries from the other.

**Speed, on-box**, 3 interleaved reps per host (range across reps):

| workload | nh3-ml1 | esh-ml1 |
|---|---|---|
| embed 1 short query, p50 | 6.60–6.74 ms | 6.97–7.02 ms |
| bulk embed, 64 per request, passages/s | 103.7–105.0 | 105.5–108.1 |
| rerank 20 docs, p50 | 155.8–159.2 ms | 151.8–160.5 ms |

The two hosts run at the same speed. esh-ml1 was also holding the idle reward seat
(~8 GB VRAM, 0% util) during these runs.

**Gateway routing (Prime's call).** The recommendation is **load-share**. The
2026-09-25 rule against load-sharing came from pairing esh-ml1 with the much
faster fv-ml1. These two cards are identical, and a second site removes the
single-host outage the esh-ml1 README warns about.

## How it is built

1. [`playbooks/pve-nvidia-host.yaml`](../../playbooks/pve-nvidia-host.yaml) on
   nh3-pve: driver **580.178.04** (open modules, DKMS) plus the
   `nvidia-persistenced` unit. It needs **Secure Boot off**, which was turned off
   in the BIOS on the 2026-09-25 visit; the pre-flight refuses otherwise.
2. [`playbooks/gpu-lxc.yaml`](../../playbooks/gpu-lxc.yaml) with the
   "Run (nh3-ml1)" `--var` line from its header.
   ⚠ nh3-pve was on **lxc-pve 6.0.0-1**. With it, every `docker run` in the CT
   failed with *"open sysctl net.ipv4.ip_unprivileged_port_start file: reopen
   fd 8: permission denied"* (runc 1.5 against the old AppArmor profile).
   Upgrading lxc-pve alone to 6.0.0-2 (Proxmox fix #7006) and then running
   `pct reboot 109` fixed it. The playbook now does the upgrade as its first step.
3. `scripts/deploy-stack.sh nh3-ml1 embed-rerank`, then set `HOST_NAME` /
   `HOST_IP` in `.env`, then `docker compose up -d`.

The **driver version lock** and the DKMS/kernel notes in
[esh-ml1's README](../esh-ml1/README.md#-driver-version-lock) apply here unchanged.
nh3-pve runs kernel 6.8.12-43, which the 2026-09-25 headers install pulled in
(see `servers/nh3-pve/README.md`).

## Monitoring and telemetry (wired 2026-09-25)

| layer | what | where |
|---|---|---|
| **Beszel** | NVIDIA agent `henrygd/beszel-agent-nvidia:0.18.7`, `stacks/beszel` + `hosts/nh3-ml1.yaml`, hub system `1feeeq61g4mkqre`; GPU util, VRAM and power are sampled | alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m (**GPU + NVMe only**: the agent drops the host's `coretemp_*`/`acpitz` with `SENSORS`, because an LXC reads nh3-pve's CPU sensors and the first alert, 2026-09-25 2105, was nh3-pve's CPU during vzdump). nh3-pve's own system carries the CPU alert at >95 °C |
| **Uptime Kuma** | `Embed — Qwen3 0.6B (TEI, nh3-ml1)` → `:8001/health` (#29); `Rerank — bge-v2-m3 (TEI, nh3-ml1)` → `:8013/health` (#30) | `stacks/uptimekuma/monitors.yaml` |
| **Homepage** | two cards under *AI - Eval & Retrieval*; dockerd on tcp/2375 bound to `10.100.50.80` | `stacks/homepage/conf/docker.yaml` → `nh3-ml1-docker` |
| **Dozzle** | agent `v10.4.1` on `10.100.50.80:7007`, compose dir `dozzle-agent`; added to the hub's `DOZZLE_REMOTE_AGENT` | hub on ana-docker :8088 |