vh
50c85e0e8a
- qwen3-embedding and reranker: second deployment on nh3-ml1 (config); reranker-a3-bge-v2-m3: second DB deployment via /model/new. - router_settings.enable_weighted_failover: true. Only the three multi-deployment groups are affected. - Measured by stopping nh3-ml1 TEI: without failover 7/40 embeds 500'd; with it rerank 80/80, embed 38/40 at onset and 60/60 over a 34 s outage; nh3 rejoins rotation after restart. Split 12/28 embed, 22/18 rerank.
137 lines
7.7 KiB
Markdown
137 lines
7.7 KiB
Markdown
# nh3-ml1
|
||
|
||
GPU LXC for the NH3 site: **CT 109 on nh3-pve**, holding the **NVIDIA RTX 2000E
|
||
Ada** (16 GB, 50 W, `01:00.0`, PCIe gen4 x8). **It is the second embedding and
|
||
reranking backend**, the twin of [esh-ml1](../esh-ml1/README.md): same card, same
|
||
driver, same TEI image and models. Built 2026-09-25, after the NH3 site visit
|
||
turned Secure Boot off on nh3-pve.
|
||
|
||
**Load-shared with esh-ml1 since 2026-09-26 (Prime).** The gateway splits
|
||
`qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3` across both hosts at
|
||
random, one `simple-shuffle` deployment each. Measured with 40 + 40 requests: embed
|
||
12 esh / 28 nh3, rerank 22 / 18.
|
||
|
||
**Failover, measured by stopping this host's TEI** (2026-09-26 0747–0753):
|
||
- Without failover settings: **7/40 embeds returned 500**. The router's default 2
|
||
retries re-picked at random and hit the dead backend again.
|
||
- With `router_settings.enable_weighted_failover: true`, which only affects
|
||
multi-deployment groups:
|
||
- rerank **80/80** OK.
|
||
- embeds **38/40** in the first test (2 failures at outage onset) and **60/60**
|
||
across a 34 s sustained outage.
|
||
- Once TEI restarted, nh3 was back in rotation (15/30).
|
||
- Residual: in LiteLLM 1.97 the embedding path does not stamp the failed
|
||
deployment, so embeds rely on the router's cooldown rather than per-request
|
||
re-pick. Expect a couple of failed embeds when a backend first dies. n = 2
|
||
outages.
|
||
|
||
| | |
|
||
|---|---|
|
||
| **IP** | `10.100.50.80/24`, VLAN 50 (`nh3-servers`), gateway `10.100.50.1` (static, outside the UDM's `.150–.249` DHCP pool) |
|
||
| **DNS** | `nh3-ml1.nh3.internal` |
|
||
| **SSH** | `ssh nh3-ml1` → `infra-ops@10.100.50.80` (NOPASSWD sudo) · from the host: `pct enter 109` |
|
||
| **OS** | Debian 12, unprivileged, `nesting=1,keyctl=1` |
|
||
| **Size** | 6 cores, 16 GB RAM + 2 GB swap, 80 GB rootfs on `local-zfs` |
|
||
| **Boot** | `onboot: 1`, `startup: order=30`, after the site's core guests, so a GPU fault never delays NH3's DNS or mesh route |
|
||
| **Backups** | **None, on purpose.** nh3-pve's vzdump job is `all 1`; the playbook added 109 to its `exclude` list. Everything is rebuilt from the playbooks and the stack; models re-download. |
|
||
|
||
## What it serves
|
||
|
||
`stacks/embed-rerank` (`/opt/docker/compose/embed-rerank`), **TEI 1.9.4**
|
||
(`89-1.9.4`). The live `.env` differs from `.env.example` only in
|
||
`HOST_NAME=nh3-ml1` and `HOST_IP=10.100.50.80`, which feed the Homepage labels.
|
||
|
||
| container | model | port |
|
||
|---|---|---|
|
||
| `tei-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 (`/v1/embeddings`, `/embed`) |
|
||
| `tei-rerank` | `BAAI/bge-reranker-v2-m3` | 8013 (`/rerank`, body `query` + `texts`) |
|
||
|
||
VRAM ~2.7 GB for both, so ~13 GB is free.
|
||
|
||
**Dataset-foundry utility seats (2026-09-26, brokkr; operator-approved):**
|
||
- `stacks/lfm-vl-seat`: LFM2.5-VL-3B on llama.cpp, `:8030`, gateway `lfm25-vl-3b`.
|
||
- `stacks/vibevoice-asr-seat`: VibeVoice-ASR-Streaming-1.5B on audio.cpp,
|
||
`:8031`, direct only.
|
||
|
||
GPU total with TEI is ~7.5 of 16 GB. Both are batch workloads, which suits this
|
||
card.
|
||
|
||
**Tried and removed (2026-09-25):** a copy of the code-completion seat
|
||
(Qwen2.5-Coder-1.5B). It gave the same quality but ran about 5× slower than on
|
||
fv-ml1 (a 64-token FIM completion took ~1.0 s vs ~0.2 s), so it stays on fv-ml1
|
||
(Prime). The recipe and measurements are in `stacks/coder-seat/README.md`.
|
||
**Lesson for the next candidate:** this card has about a fifth of a Blackwell's
|
||
decode speed on small generative models, so it suits embedding, reranking and
|
||
classification, not latency-sensitive generation.
|
||
|
||
## Parity and speed vs esh-ml1 (2026-09-25 ~1540 PT)
|
||
|
||
Client on nh3-dev, calling both seats directly. Corpus: 1,120 paragraphs from
|
||
this repo's docs plus 6 fixed texts (CJK, code, emoji, a 1-char input, a 6k-char
|
||
passage); 50 instruction-format queries; each host embedded everything twice.
|
||
|
||
| embedding check | result | noise floor / control |
|
||
|---|---|---|
|
||
| per-text cosine, esh vs nh3 | median 0.999998, min 0.999993 | esh vs esh 0.999998 / 0.999995; nh3 vs nh3 0.999998 / 0.999993 |
|
||
| overlap@10, nh3 queries on the esh index | 1.000 | esh rerun 1.000; **positive control** MRL-256 truncation 0.684 |
|
||
| overlap@10, nh3 index + nh3 queries | 1.000 | — |
|
||
| hit@1 own paragraph | 0.76 (both hosts) | — |
|
||
| **negative control**, different texts | cosine median 0.50 | — |
|
||
|
||
| rerank check (100 queries × 20 docs) | esh vs nh3 | esh vs esh | **positive control** (query cut to 4 words) |
|
||
|---|---|---|---|
|
||
| top-1 agreement | 1.00 | 1.00 | 0.94 |
|
||
| top-5 exact order | 0.98 | 0.97 | 0.09 |
|
||
| max score difference | 0.0014 | 0.0020 | 0.97 |
|
||
|
||
**The two hosts cannot be told apart.** Every esh-vs-nh3 figure sits inside the
|
||
esh-vs-esh noise. Sensitivity floor: this method cannot resolve an embedding
|
||
cosine gap below ~5×10⁻⁶ or a rerank score difference below ~0.002. An index
|
||
built on either host serves queries from the other.
|
||
|
||
**Speed, on-box**, 3 interleaved reps per host (range across reps):
|
||
|
||
| workload | nh3-ml1 | esh-ml1 |
|
||
|---|---|---|
|
||
| embed 1 short query, p50 | 6.60–6.74 ms | 6.97–7.02 ms |
|
||
| bulk embed, 64 per request, passages/s | 103.7–105.0 | 105.5–108.1 |
|
||
| rerank 20 docs, p50 | 155.8–159.2 ms | 151.8–160.5 ms |
|
||
|
||
The two hosts run at the same speed. esh-ml1 was also holding the idle reward seat
|
||
(~8 GB VRAM, 0% util) during these runs.
|
||
|
||
**Gateway routing (Prime's call).** The recommendation is **load-share**. The
|
||
2026-09-25 rule against load-sharing came from pairing esh-ml1 with the much
|
||
faster fv-ml1. These two cards are identical, and a second site removes the
|
||
single-host outage the esh-ml1 README warns about.
|
||
|
||
## How it is built
|
||
|
||
1. [`playbooks/pve-nvidia-host.yaml`](../../playbooks/pve-nvidia-host.yaml) on
|
||
nh3-pve: driver **580.178.04** (open modules, DKMS) plus the
|
||
`nvidia-persistenced` unit. It needs **Secure Boot off**, which was turned off
|
||
in the BIOS on the 2026-09-25 visit; the pre-flight refuses otherwise.
|
||
2. [`playbooks/gpu-lxc.yaml`](../../playbooks/gpu-lxc.yaml) with the
|
||
"Run (nh3-ml1)" `--var` line from its header.
|
||
⚠ nh3-pve was on **lxc-pve 6.0.0-1**. With it, every `docker run` in the CT
|
||
failed with *"open sysctl net.ipv4.ip_unprivileged_port_start file: reopen
|
||
fd 8: permission denied"* (runc 1.5 against the old AppArmor profile).
|
||
Upgrading lxc-pve alone to 6.0.0-2 (Proxmox fix #7006) and then running
|
||
`pct reboot 109` fixed it. The playbook now does the upgrade as its first step.
|
||
3. `scripts/deploy-stack.sh nh3-ml1 embed-rerank`, then set `HOST_NAME` /
|
||
`HOST_IP` in `.env`, then `docker compose up -d`.
|
||
|
||
The **driver version lock** and the DKMS/kernel notes in
|
||
[esh-ml1's README](../esh-ml1/README.md#-driver-version-lock) apply here unchanged.
|
||
nh3-pve runs kernel 6.8.12-43, which the 2026-09-25 headers install pulled in
|
||
(see `servers/nh3-pve/README.md`).
|
||
|
||
## Monitoring and telemetry (wired 2026-09-25)
|
||
|
||
| layer | what | where |
|
||
|---|---|---|
|
||
| **Beszel** | NVIDIA agent `henrygd/beszel-agent-nvidia:0.18.7`, `stacks/beszel` + `hosts/nh3-ml1.yaml`, hub system `1feeeq61g4mkqre`; GPU util, VRAM and power are sampled | alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m (**GPU + NVMe only**: the agent drops the host's `coretemp_*`/`acpitz` with `SENSORS`, because an LXC reads nh3-pve's CPU sensors and the first alert, 2026-09-25 2105, was nh3-pve's CPU during vzdump). nh3-pve's own system carries the CPU alert at >95 °C |
|
||
| **Uptime Kuma** | `Embed — Qwen3 0.6B (TEI, nh3-ml1)` → `:8001/health` (#29); `Rerank — bge-v2-m3 (TEI, nh3-ml1)` → `:8013/health` (#30) | `stacks/uptimekuma/monitors.yaml` |
|
||
| **Homepage** | two cards under *AI - Eval & Retrieval*; dockerd on tcp/2375 bound to `10.100.50.80` | `stacks/homepage/conf/docker.yaml` → `nh3-ml1-docker` |
|
||
| **Dozzle** | agent `v10.4.1` on `10.100.50.80:7007`, compose dir `dozzle-agent`; added to the hub's `DOZZLE_REMOTE_AGENT` | hub on ana-docker :8088 |
|