feat(esh-ml1): RTX 2000E Ada on esh-pve serves embed + rerank as a LiteLLM failover
esh-pve: NVIDIA 580.178.04 (open modules, DKMS) installed on the host from NVIDIA's .run and loaded live, no reboot. nvidia-persistenced unit creates the device nodes before pve-guests; the T400's vfio-pci ids and `blacklist nvidia` retired. playbooks/esh-pve-nvidia-host.yaml. esh-ml1: CT 110, unprivileged Debian 12, 10.0.50.80, GPU nodes via devN, NVIDIA userspace from the same .run (--no-kernel-modules), docker-ce + nvidia-container-toolkit (no-cgroups). playbooks/esh-ml1-lxc.yaml. Not in the vzdump job on purpose. DNS esh-ml1.esh.internal. stacks/embed-rerank: Qwen3-Embedding-0.6B :8001 + bge-reranker-v2-m3 :8013 on the same vLLM v0.24.0 digest and flags as fv-ml1. Measured parity: embed cosine FV-vs-ESH median 0.999908 (min 0.999772), inside both self-noise floors; rerank max |delta| 0.000145 vs floor 0.000181, identical ranking. litellm: qwen3-embedding and reranker gain an esh-ml1 deployment at order 2 behind fv-ml1 (order 1). Order fallback proven with throwaway groups: refused primary +0.15 s, host-down primary ~18.7 s per call, dead-only 500. Also: repaired the DB-only alias reranker-a3-bge-v2-m3, dead since the fv-ml1 relocation (still named 10.250.50.54); documented the third unkillable homepage wedge on esh-docker-vm.
This commit is contained in:
@@ -73,3 +73,32 @@ Like `nh3-docker`, this host runs **Dozzle** and **Beszel** agents that report b
|
||||
## Placement rule
|
||||
|
||||
Home-lab workloads for the ESH site go here. Not part of the PFI colo topology.
|
||||
|
||||
## ⚠ Recurring: `homepage` wedges unkillably (3× — 2026-06-03, 2026-09-18, 2026-09-24)
|
||||
|
||||
The Homepage container stops answering (Uptime Kuma: `timeout of 16000ms
|
||||
exceeded`; healthcheck `Connecting to 127.0.0.1:3000` times out) and cannot be
|
||||
stopped. Signature captured 2026-09-24 ~2224 PT, guest kernel `6.1.0-41-amd64`:
|
||||
|
||||
- one `node` thread in **D state in `vm_mmap_pgoff`** — waiting for its own
|
||||
process's `mmap_lock` for write, with **no visible holder** (every other
|
||||
thread sat in `futex_wait`; a scan of every task's kernel stack found no
|
||||
reader in a fault, NFS or `access_remote_vm` path except the `ps` calls
|
||||
queued behind it).
|
||||
- `docker restart` → *"tried to kill container, but did not receive an exit
|
||||
event"*; the process then sits in **`exit_mmap`** (uninterruptible) with
|
||||
PID 1 of the container in `zap_pid_ns_processes`. The June entry in
|
||||
archival-memory records the same `exit_mmap` end state.
|
||||
- The rest of the VM is fine: DNS (AdGuard) answered, 17 other containers up.
|
||||
|
||||
⚠ **Diagnosing it can hang your shell.** `ps`, `pgrep` and `docker top` read
|
||||
`/proc/<pid>/cmdline|environ`, which takes the same lock, so they block in D
|
||||
state too. Read `/proc/<pid>/task/*/stat` and `sudo cat .../stack` instead
|
||||
(`ssh infra-ops@10.0.50.45`, NOPASSWD).
|
||||
|
||||
**Only a VM reboot clears it**, and this VM is ESH's DNS resolver, so the
|
||||
reboot is a short ESH-wide DNS outage — schedule it. Root cause not
|
||||
established; a kernel-side mmap_lock problem is the leading guess, not a
|
||||
finding. Moving Homepage to ana-docker (as was done for Uptime Kuma on
|
||||
2026-09-21 for the same box's history) would take the dashboard out of this
|
||||
failure domain.
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
# esh-ml1
|
||||
|
||||
GPU LXC for the ESH home lab: **CT 110 on esh-pve**, holding the **NVIDIA RTX
|
||||
2000E Ada** (16 GB, 50 W, `01:00.0`). It serves the fleet's embedding and
|
||||
reranking models locally at ESH. Built 2026-09-24.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **IP** | `10.0.50.80/24`, VLAN 50, gateway `10.0.50.1` (static, outside the UDM's `.150–.250` DHCP pool) |
|
||||
| **DNS** | `esh-ml1.esh.internal` |
|
||||
| **SSH** | `ssh esh-ml1` → `infra-ops@10.0.50.80` (NOPASSWD sudo) · from the host: `pct enter 110` |
|
||||
| **OS** | Debian 12, unprivileged, `nesting=1,keyctl=1` |
|
||||
| **Size** | 6 cores, 16 GB RAM + 2 GB swap, 80 GB rootfs on `local-lvm` |
|
||||
| **Boot** | `onboot: 1`, `startup: order=30` — after esh-scale (1), esh-vm-db (10) and esh-vm-docker (20), so a GPU fault never delays ESH's DNS or mesh route |
|
||||
| **Backups** | **None, on purpose.** esh-pve's vzdump job lists vmids explicitly and 110 is not one. Everything is rebuilt from the playbooks and the stack; models re-download. |
|
||||
|
||||
## What it serves
|
||||
|
||||
`stacks/embed-rerank` (`/opt/docker/compose/embed-rerank`):
|
||||
|
||||
| container | model | port | gateway name |
|
||||
|---|---|---|---|
|
||||
| `vllm-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 | `qwen3-embedding` (order 2) |
|
||||
| `vllm-rerank-bge` | `BAAI/bge-reranker-v2-m3` | 8013 | `reranker` (order 2) |
|
||||
|
||||
The same models, vLLM version (`v0.24.0`, digest `251eba5cc7c1`) and flags as
|
||||
fv-ml1's `vllm` stack, so the two sites are interchangeable. In LiteLLM they
|
||||
are the **order-2 failover** behind fv-ml1: fv-ml1 serves every request while
|
||||
it is up.
|
||||
|
||||
**Parity, measured 2026-09-24** (11 texts incl. CJK, code, a 6k-char passage;
|
||||
2 runs per site):
|
||||
|
||||
| | median | min |
|
||||
|---|---|---|
|
||||
| embed cosine FV vs ESH, same text | 0.999908 | 0.999772 |
|
||||
| noise floor FV vs FV | 0.999927 | 0.999791 |
|
||||
| noise floor ESH vs ESH | 0.999911 | 0.999809 |
|
||||
| negative control, different texts | 0.232 | 0.071 |
|
||||
|
||||
The cross-site difference is inside each site's own run-to-run noise; this
|
||||
method cannot resolve a cosine gap below ~2×10⁻⁴. Reranker scores differed by
|
||||
at most 0.000145 (FV-vs-FV floor 0.000181), with identical ranking.
|
||||
|
||||
**VRAM:** 0.20 × 16,380 MiB each; 4,823 MiB in use with both loaded, ~11 GB
|
||||
free.
|
||||
|
||||
## How it is built
|
||||
|
||||
1. [`playbooks/esh-pve-nvidia-host.yaml`](../../playbooks/esh-pve-nvidia-host.yaml)
|
||||
— driver **580.178.04** (open modules, DKMS) on the **hypervisor**, the
|
||||
`nvidia-persistenced` unit that creates the device nodes before
|
||||
`pve-guests`, and removal of the old VFIO/blacklist config.
|
||||
2. [`playbooks/esh-ml1-lxc.yaml`](../../playbooks/esh-ml1-lxc.yaml) — the CT,
|
||||
`dev0–3` GPU nodes, the NVIDIA userspace from the **same `.run`** with
|
||||
`--no-kernel-modules`, fleet ids (infra-ops 850, docker 851, vh 1000),
|
||||
docker-ce and nvidia-container-toolkit with `no-cgroups = true`.
|
||||
3. `scripts/deploy-stack.sh esh-ml1 embed-rerank`, then `docker compose up -d`.
|
||||
|
||||
Both playbooks are idempotent; re-run them to repair.
|
||||
|
||||
## ⚠ Driver version lock
|
||||
|
||||
The kernel module lives on esh-pve; the libraries live in this container. They
|
||||
**must be the same version**, or every CUDA call fails with *"driver/library
|
||||
version mismatch"*. To upgrade: bump `driver_version` + `driver_sha256` in the
|
||||
host playbook and `driver_version` in the LXC playbook, run the host one, then
|
||||
the LXC one, then restart the stack.
|
||||
|
||||
A PVE kernel update is handled by DKMS (`proxmox-headers-6.8` pulls headers
|
||||
for each new kernel). Moving esh-pve to a different kernel series (6.14 opt-in)
|
||||
needs that series' headers meta-package installed first, or the module will not
|
||||
build and this CT will fail to start at the next boot.
|
||||
|
||||
## Not yet wired
|
||||
|
||||
- **Homepage**: the compose carries labels, but esh-ml1 is not in
|
||||
`stacks/homepage/conf/docker.yaml` (it would need dockerd on tcp/2375 like
|
||||
the other hosts).
|
||||
- **Beszel**: no agent yet.
|
||||
- **ESH consumers** (Open WebUI RAG, Paperless) still go through the gateway
|
||||
at ana-docker, so they do not survive a mesh outage. See persistent-memory
|
||||
for the open decision.
|
||||
@@ -0,0 +1 @@
|
||||
infra-ops@10.0.50.80
|
||||
File diff suppressed because one or more lines are too long
@@ -13,7 +13,8 @@ Proxmox VE hypervisor for the ESH home lab (`pve.esteban.net`). Non-PFI scope.
|
||||
|
||||
- **CPU:** 13th Gen Intel Core i9-13900H
|
||||
- **RAM:** 62.5 GB
|
||||
- **Kernel:** `6.8.12-16-pve` (Proxmox 8.x)
|
||||
- **Kernel:** `6.8.12-42-pve`, PVE 8.4.20 (as of 2026-09-24)
|
||||
- **GPU:** NVIDIA RTX 2000E Ada, 16 GB (`01:00.0`) — driven by the **host** (580.178.04, DKMS), used by the esh-ml1 LXC. See below.
|
||||
- **Storage:** local `pve-root` at 94G (21% used) + two large NFS mounts from `10.0.50.50`:
|
||||
- `/mnt/pve/esh-nas` — 92 TB available on `/mnt/pvestore`
|
||||
- `/mnt/pve/tank-vmbu` — 93 TB, VM backup target (~1.4 TB used)
|
||||
@@ -47,10 +48,39 @@ fully up. esh-vm-docker, esh-vm-db and esh-scale all started by themselves.
|
||||
esh-vm-docker's NFS (`/mnt/books`, `/mnt/backup`) mounted, traefik served with no
|
||||
403s, and AdGuard resolved.
|
||||
|
||||
The new card has **no driver bound**. `/etc/modprobe.d` still lists the T400's IDs
|
||||
for vfio-pci (`10de:1ff2,10de:10fa`), which no longer match anything; harmless, but
|
||||
clean it up when the card's use is decided. The intended use is a local
|
||||
embedding endpoint.
|
||||
The new card got its driver the same night — see the next section.
|
||||
|
||||
## GPU — host driver + the esh-ml1 LXC (2026-09-24)
|
||||
|
||||
**Prime's decision: LXC + host driver, NOT a VFIO VM.** Both hangs on this box
|
||||
this year came from VFIO passthrough. The card now runs under the host's own
|
||||
NVIDIA driver and is shared into **CT 110 `esh-ml1`**, which serves embedding
|
||||
and reranking ([`servers/esh-ml1/README.md`](../esh-ml1/README.md)).
|
||||
|
||||
- **Driver 580.178.04, open kernel modules, DKMS**, from NVIDIA's `.run`
|
||||
(`/root/nvidia/`). Applied by
|
||||
[`playbooks/esh-pve-nvidia-host.yaml`](../../playbooks/esh-pve-nvidia-host.yaml)
|
||||
**live, with no reboot**: nouveau was never loaded and nothing held the card.
|
||||
- **`nvidia-persistenced.service`** (ours, in `/etc/systemd/system`) runs
|
||||
`nvidia-modprobe -c0 -u` and the persistence daemon **before
|
||||
`pve-guests`**, so `/dev/nvidia0`, `nvidiactl` and `nvidia-uvm{,-tools}`
|
||||
exist when CT 110 starts. Without them `pct start` refuses the `devN:`
|
||||
entries. If the unit fails, only CT 110 fails; no other guest depends on it.
|
||||
- `/etc/modules-load.d/nvidia.conf` loads `nvidia` + `nvidia_uvm`.
|
||||
- **Retired:** the T400's vfio-pci id binding (`/etc/modprobe.d/vfio.conf`, moved
|
||||
to `/root/nvidia/vfio.conf.retired-2026-09-24`) and `blacklist nvidia` in
|
||||
`/etc/modprobe.d/blacklist.conf` (`blacklist nouveau` stays). The `vfio*`
|
||||
lines in `/etc/modules` were left alone; they bind nothing. The initramfs was
|
||||
not rebuilt, to keep the boot path untouched; the next kernel update will.
|
||||
- The installer also enabled NVIDIA's `nvidia-{suspend,hibernate,resume}`
|
||||
units. They only act on a suspend and are harmless on a server.
|
||||
- ⚠ **Kernel updates:** DKMS rebuilds the module when a new
|
||||
`proxmox-headers-6.8.*` arrives (the `proxmox-headers-6.8` meta-package is
|
||||
installed). Before opting into a different kernel series, install that
|
||||
series' headers meta-package, or the next boot comes up with no GPU and CT
|
||||
110 will not start.
|
||||
- ⚠ **The host module and CT 110's libraries must stay the same version.**
|
||||
Upgrade both playbooks together (host first).
|
||||
|
||||
## Watchdog — hardware, not software
|
||||
|
||||
|
||||
Reference in New Issue
Block a user