diff --git a/CLAUDE.md b/CLAUDE.md index 0112337..75e7609 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -362,6 +362,7 @@ Observed and standardized across servers: | esh-pve | 10.0.250.35 | ESH home lab | Proxmox VE hypervisor | `servers/esh-pve/README.md` | | esh-pve-nas | 10.0.50.55 | ESH home lab | Proxmox VE hypervisor (storage / media) | `servers/esh-pve-nas/README.md` | | esh-vm-db | 10.0.50.60 | ESH home lab | DB VM — PostgreSQL (paperless-ng) + MongoDB; bare-metal VM, no Docker | `servers/esh-vm-db/README.md` | +| esh-ml1 | 10.0.50.80 | ESH home lab | GPU LXC (CT 110 on esh-pve, RTX 2000E Ada 16 GB, host-side NVIDIA driver) — embedding + reranking, LiteLLM order-2 failover behind fv-ml1 | `servers/esh-ml1/README.md` | **Placement rules:** - GPU-required stacks → `fv-ml1` (primary, Fountain Valley) or `irv-ml1` diff --git a/dns/internal.yaml b/dns/internal.yaml index 6375ed6..eab1458 100644 --- a/dns/internal.yaml +++ b/dns/internal.yaml @@ -99,6 +99,7 @@ hosts: - {name: esh-pve, site: esh, v4: 10.0.250.35, note: Proxmox hypervisor} - {name: esh-pve-nas, site: esh, v4: 10.0.50.55, note: Proxmox hypervisor — storage/media} - {name: esh-vm-db, site: esh, v4: 10.0.50.60, note: PostgreSQL + MongoDB} + - {name: esh-ml1, site: esh, v4: 10.0.50.80, note: CT110 on esh-pve — RTX 2000E Ada, embedding + reranking (vLLM)} - {name: vm-esh-nas, site: esh, v4: 10.0.50.154, note: NAS-adjacent docker host} - {name: esh-filebot, site: esh, v4: 10.0.50.70, note: restic / file-sync VM} - {name: esh-gw, site: esh, v4: 10.0.250.1, note: esh-gw} diff --git a/docs/pfi/reranker-selection-ledger.md b/docs/pfi/reranker-selection-ledger.md index 909d4f9..2015f21 100644 --- a/docs/pfi/reranker-selection-ledger.md +++ b/docs/pfi/reranker-selection-ledger.md @@ -228,3 +228,18 @@ on fv-ml1; `/model/delete` the two aliases (DB-persisted); weights retained in H (recreate = brief blip), (c) run a 2nd A3 replica load-balanced behind `reranker` (~2× tput, identical replicas so no split-measurement issue now the bake-off is closed). Brokkr will re-run the grid against A4 if it bites — no intuition swaps. + +## 2026-09-24 — second site + a dead alias repaired (infra-ops) + +- **`reranker-a3-bge-v2-m3` was dead from the fv-ml1 relocation (2026-09-12) until + 2026-09-24.** It is a DB-only alias (added via /model/new, so not in `config.yaml`) and + still named ana-ml2's old address `10.250.50.54:8013`; every call returned 500 after ~23 s. + Spend logs show **no callers in the 7-day retention window** besides the probe that found + it. Repointed with `PATCH /model/{id}/update` to `10.251.50.54:8013` (3/3 × 200). Lesson: + a DB-only alias is invisible to a grep of the config, so an IP migration misses it — + sweep `/model/info` too. +- **esh-ml1 added as an order-2 failover behind `reranker`** (and `qwen3-embedding`) — same + model and vLLM version on an RTX 2000E Ada at ESH. This is lever (c) above in failover + form, not load-balancing: fv-ml1 serves every request while it is up. Parity: |score| + FV-vs-ESH max 0.000145 vs FV-vs-FV floor 0.000181, identical ranking over 11 documents. + See `servers/esh-ml1/README.md`. diff --git a/persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md b/persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md new file mode 100644 index 0000000..75381c7 --- /dev/null +++ b/persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md @@ -0,0 +1,62 @@ +# `[2026-09-24]` esh-ml1 built: RTX 2000E Ada serving embed + rerank, LiteLLM order-2 failover behind fv-ml1 + +Executes Prime's 2026-09-24 decision (LXC + host driver, not a VFIO VM). Done +2204–2240 PT the same night, **with no esh-pve reboot**. + +**Host (esh-pve).** NVIDIA **580.178.04** open modules via DKMS from NVIDIA's +`-no-compat32.run` (sha256 checked against NVIDIA's published sum), applied +live by `playbooks/esh-pve-nvidia-host.yaml`: nouveau was never loaded and the +T400's vfio ids matched nothing, so nothing held the card. Our own +`nvidia-persistenced.service` creates `/dev/nvidia{0,ctl,-uvm,-uvm-tools}` +**before `pve-guests`** (else `pct start` refuses the `devN:` entries). Retired +`vfio.conf` (moved to `/root/nvidia/`) and `blacklist nvidia`. Initramfs NOT +rebuilt on purpose (boot path untouched). 580 because the fleet's vLLM +`v0.24.0` image is CUDA 13.0 (driver ≥ 580); fv-ml1 runs 580.65.06. `.run` +not an apt repo: one file on both sides makes the host-module/container-lib +version match true by construction; apt would let an upgrade move one side. + +**CT 110 `esh-ml1`** (`playbooks/esh-ml1-lxc.yaml`): unprivileged Debian 12, +nesting+keyctl, 6c/16G/80G `local-lvm`, `10.0.50.80` VLAN 50 (static; UDM pool +is `.150–.250`), `startup order=30` after esh-scale/VMs. NVIDIA userspace from +the same `.run` with `--no-kernel-modules`; docker-ce; nvidia-container-toolkit +with `no-cgroups=true`. **Docker-in-unprivileged-LXC with the GPU worked first +try on lxc-pve 6.0.0-2** (no AppArmor workaround needed). Fleet ids per +`docs/pfi/fleet-conventions.md` (infra-ops 850, docker 851, vh 1000 — vh has +no key/password yet). **Not in the vzdump job, on purpose** (rebuildable). +DNS `esh-ml1.esh.internal` synced to all three resolvers. + +**Stack `stacks/embed-rerank`:** `vllm-embed` :8001 + `vllm-rerank-bge` :8013, +same models/flags/vLLM digest (`251eba5cc7c1`) as fv-ml1, 0.20 mem-util each → +4,823 MiB used of 16,380. + +**Parity (the load-bearing measurement; embeddings are model-specific).** 11 +texts × 2 runs per site. Embed cosine FV-vs-ESH median 0.999908 / min 0.999772, +inside both self-noise floors (FV-vs-FV 0.999927 / 0.999791; ESH-vs-ESH +0.999911 / 0.999809). Negative control (different texts) 0.07–0.38. Sensitivity +floor ~2×10⁻⁴ cosine. Rerank |score| max 0.000145 vs FV-FV floor 0.000181, +identical ranking. + +**Gateway.** `qwen3-embedding` and `reranker` each got a second deployment +(`order: 2`, esh-ml1); fv-ml1 is `order: 1`. LiteLLM v1.97 order-fallback +proven with throwaway groups (created + deleted): primary refusing → ESH in ++0.15 s (3/3); primary host-down (no ARP) → ESH but **~18.7 s on every call, +no cooldown** (3/3); dead-only negative control → 500 (3/3). Post-restart +production traffic is served by fv-ml1 (header `x-litellm-model-api-base`). +`/health?model=` returned 503 with empty lists for both groups — not +investigated. Gateway restart: liveliness back in ~52 s. + +**Found + fixed:** DB-only alias `reranker-a3-bge-v2-m3` pointed at ana-ml2's +pre-relocation IP `10.250.50.54` — dead since 2026-09-12 (500 after ~23 s), no +callers in 7-day spend logs. PATCHed to `10.251.50.54:8013`. Sweep of +`/model/info` found no other `10.250.50.54` targets. + +**Who actually uses these names (spend logs, 7 days to 2026-09-25 05:35Z):** +`qwen3-embedding` — worldtree-gateway 481 (nh3-dev + corviduo-dev), nevermore +20; `reranker` — nevermore 13. **No ESH-side consumer called either in 7 days** +(Open WebUI's RAG embeds only on document upload). So today esh-ml1's value is +an FV-outage failover for Worldtree + nevermore, not local service for ESH. + +**Open:** a direct ESH→esh-ml1 path (survives a mesh outage) would help only +ESH consumers, which are idle — recommended parked. esh-ml1 not yet in +Homepage docker.yaml or Beszel. The ~18.7 s host-down failover penalty is +untuned (router retries/cooldown). diff --git a/persistent-memory.md b/persistent-memory.md index 5e41f3f..a7b65a5 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-24 ~2150 PT (NH3 power outage recovered: pbs-nh3 onboot set, NFS → automount. esh-pve: VM 102 retired, a 14-day hung VFIO process cleared, T400 → RTX 2000 Ada; ⭐ NEXT = provision it as LXC + host driver for embed/rerank. pfi-gx10 AC-restore patch applied but UNVALIDATED — box OFF until Prime's AC pull 09-25. elway root:root fix + fleet ownership audit. Miranda standing order + Prime callsign in CLAUDE.md. task-board mothballed.)_ +_Last updated: 2026-09-24 ~2245 PT (⭐ esh-ml1 BUILT: RTX 2000E Ada as CT 110 on esh-pve, host driver 580.178.04 live-loaded with NO reboot, vLLM embed+rerank at parity with fv-ml1, LiteLLM order-2 failover. Dead `reranker-a3-bge-v2-m3` alias repaired. ⚠ homepage wedged unkillably on esh-docker-vm (3rd time) — needs a VM reboot, Prime's call. pfi-gx10 AC-restore still UNVALIDATED — AC pull 09-25.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -115,30 +115,26 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-09-24 ~2150 PT._ +_As of 2026-09-24 ~2245 PT._ -### ⭐ NEXT: provision the RTX 2000 Ada on esh-pve as an LXC + host NVIDIA driver +### ✅ esh-ml1 built (2026-09-24) — two follow-ups -Prime's decision (2026-09-24, over a VFIO VM): the card serves **embedding + -reranking offload**. Installed tonight in esh-pve's single slot (`01:00.0`, -`10de:28b0`), no driver bound. Plan shape, not yet started: +The RTX 2000E Ada now serves `qwen3-embedding` + `reranker` from **CT 110 +`esh-ml1`** (10.0.50.80) as the **order-2 failover** behind fv-ml1. Host driver +went on live, no esh-pve reboot. Full record → Recent decisions. +- **Parked unless Prime wants it:** a direct ESH→esh-ml1 path for ESH consumers + (survives a mesh outage). Spend logs show NO ESH-side caller in 7 days; the + real users are worldtree-gateway + nevermore. +- **Not wired yet:** esh-ml1 in Homepage `docker.yaml` (needs dockerd tcp/2375) + and Beszel. The ~18.7 s host-down failover penalty is untuned. -1. NVIDIA driver on the PVE host (headers for the running `6.8.12-*-pve` - kernel; DKMS). Blacklist nouveau. Remove the stale T400 vfio ids from - `/etc/modprobe.d` (`10de:1ff2,10de:10fa`). -2. An LXC (Debian 12 template — PVE here rejects Debian 13) with the GPU - device nodes bound in and the SAME userspace driver version as the host; - docker + nvidia-container-toolkit inside. -3. Serve the SAME models the fleet already uses so vectors stay compatible: - `qwen3-embedding` (currently fv-ml1:8001 via LiteLLM) and the rerankers - (`reranker`, `reranker-a3-bge-v2-m3`). Confirm sizes from the live seats - before choosing an engine (TEI vs vLLM). -4. Wire as a LiteLLM failover/local deployment under the SAME model names; ESH - consumers (Open WebUI, Paperless) keep working if FV or the mesh is down. +### ⚠ homepage wedged on esh-docker-vm — needs a VM reboot (Prime's call) -⚠ esh-pve is ESH's only DNS and its only mesh route (esh-scale CT 108 lives -there). Driver work means reboots: do them when an ESH outage is acceptable, -and confirm power-off by the light, not by ping (my path in dies with esh-scale). +3rd unkillable wedge (06-03, 09-18, now 2026-09-24 ~2224 PT): node thread D in +`vm_mmap_pgoff`, then `exit_mmap` after a kill; `docker restart` failed. Rest +of the VM healthy. The reboot blips ESH DNS. Signature + safe diagnosis +commands: `servers/esh-docker-vm/README.md`. Option to move Homepage to +ana-docker surfaced to Prime. ### pfi-gx10 is OFF — AC-pull test 2026-09-25 (Prime) @@ -163,9 +159,6 @@ a shutdown (stays off by design). Outcomes and revert in - **infra-hermes owns a daily 0110 job** proving the first High Seat report (`~/.high-seat/reports/*.jsonl`) lands in a nh3-dev restic snapshot; it replies to svos-dev (thread `01M37P61Q85KWDVYN0A00P8856`) and pings me. -- **Credentials in auto-memory:** the new global rule says never write one into - a memory file (memory is copied off-box hourly to `vh/claude-memory`). Four - of my memory files still carry the shared LiteLLM key literal — scrub them. - **Auto-memory `MEMORY.md` is over its 24.4 KB load limit** (tail truncated at load) — shorten index lines. - ESH has a single outside route (esh-scale on esh-pve) — noted, untracked. @@ -173,6 +166,7 @@ a shutdown (stays off by design). Outcomes and revert in ## Recent decisions +- `[2026-09-24]` **esh-ml1 built — RTX 2000E Ada as CT 110 on esh-pve (host driver 580.178.04 DKMS, loaded live, no reboot), vLLM embed+rerank at measured parity with fv-ml1, wired as LiteLLM `order: 2` failover; dead DB alias `reranker-a3-bge-v2-m3` (pre-relocation IP) repaired.** → `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md` - `[2026-09-24]` **esh-pve: VM 102 retired, a 14-day hung VFIO process found and cleared, T400 → RTX 2000 Ada; Prime chose LXC + host driver for embed/rerank** (implementation deferred to next session, tracked here + `servers/esh-pve/README.md`). → `persistent-memory.d/2026-09-24-esh-pve-vm102-retired-gpu-swap.md` - `[2026-09-24]` **pfi-gx10 AC-restore UEFI patch applied but NOT validated; box OFF until Prime's AC pull 2026-09-25** (tracking: `servers/pfi-gx10/README.md`, `e5197a3`). → `persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md` - `[2026-09-24]` **NH3 power outage recovered** — pbs-nh3 had no `onboot` (set), NFS boot race fixed with automount (`1cbde50`), every other Claude session on nh3-dev died. → `persistent-memory.d/2026-09-24-nh3-power-outage-recovery.md` diff --git a/playbooks/esh-ml1-lxc.yaml b/playbooks/esh-ml1-lxc.yaml new file mode 100644 index 0000000..5e19129 --- /dev/null +++ b/playbooks/esh-ml1-lxc.yaml @@ -0,0 +1,175 @@ +# esh-ml1 — the GPU LXC on esh-pve that serves embedding + reranking on the +# RTX 2000E Ada (16 GB). Runs against the HYPERVISOR and drives the container +# through `pct`, so it works before the container has any users or keys. +# +# Prerequisite: playbooks/esh-pve-nvidia-host.yaml (host kernel module, device +# nodes, persistence unit ordered before pve-guests). +# +# WHAT IT BUILDS +# - CT {{ ctid }} `esh-ml1`, unprivileged, Debian 12 (the template the host's +# own bookworm matches), nesting+keyctl for Docker, static 10.0.50.80 on +# VLAN 50 (outside the UDM's .150–.250 DHCP pool), onboot AFTER esh-scale +# and the two VMs so a GPU fault can never delay DNS or the mesh route. +# - /dev/nvidia0, nvidiactl, nvidia-uvm, nvidia-uvm-tools bound in via +# `devN:` (PVE handles the cgroup rules and node creation). +# - NVIDIA userspace from THE SAME .run the host installed, with +# --no-kernel-modules. Host module and container libraries must be the same +# version or every CUDA call fails with "driver/library version mismatch". +# - Fleet identities per docs/pfi/fleet-conventions.md: docker gid 851 +# (created BEFORE docker-ce so the package does not pick one), infra-ops +# 850 NOPASSWD, vh 1000, /opt/docker root:docker 2775. +# - docker-ce + nvidia-container-toolkit with no-cgroups=true (an unprivileged +# LXC cannot write the device cgroup; the LXC's own devN rules already +# confine the GPU). +# +# NOT BACKED UP, on purpose: esh-pve's vzdump job lists vmids explicitly and +# this CT is not in it. Everything here is rebuilt by this playbook plus +# stacks/embed-rerank; the models re-download from Hugging Face. +# +# Run: scripts/elway root@esh-pve --playbook playbooks/esh-ml1-lxc.yaml + +vars: + ctid: 110 + hostname: esh-ml1 + ip_cidr: 10.0.50.80/24 + gateway: 10.0.50.1 + vlan: 50 + cores: 6 + memory_mb: 16384 + swap_mb: 2048 + rootfs_gb: 80 + template: debian-12-standard_12.12-1_amd64.tar.zst + # Must equal playbooks/esh-pve-nvidia-host.yaml's driver_version. + driver_version: 580.178.04 + stage_dir: /root/nvidia + infra_ops_pubkey: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN+1HBwfXrkfTYWdcnWCjLJ6VLAGC87gxH5h5vKaaA3c infra-ops@pfi-fleet" + +steps: + - name: Fetch the Debian 12 template + shell: pveam update >/dev/null && pveam download local {{ template }} + creates: /var/lib/vz/template/cache/{{ template }} + + - name: Create CT {{ ctid }} ({{ hostname }}) with the GPU device nodes bound in + shell: | + pct create {{ ctid }} local:vztmpl/{{ template }} \ + --hostname {{ hostname }} --unprivileged 1 --features nesting=1,keyctl=1 \ + --cores {{ cores }} --memory {{ memory_mb }} --swap {{ swap_mb }} \ + --rootfs local-lvm:{{ rootfs_gb }} \ + --net0 name=eth0,bridge=vmbr0,firewall=1,gw={{ gateway }},ip={{ ip_cidr }},tag={{ vlan }},type=veth \ + --onboot 1 --startup order=30,up=10 \ + --dev0 /dev/nvidia0 --dev1 /dev/nvidiactl \ + --dev2 /dev/nvidia-uvm --dev3 /dev/nvidia-uvm-tools \ + --description "esh-ml1 — RTX 2000E Ada embedding + reranking (vLLM). Built by eshpfi playbooks/esh-ml1-lxc.yaml; stack stacks/embed-rerank. NOT in the vzdump job on purpose (rebuildable)." + when: "! pct status {{ ctid }} >/dev/null 2>&1" + + - name: Start the container + shell: pct start {{ ctid }} && sleep 5 + when: "! pct status {{ ctid }} | grep -q running" + + - name: Base packages + bookworm point upgrade + shell: | + pct exec {{ ctid }} -- bash -s <<'EOF' + set -euo pipefail + export DEBIAN_FRONTEND=noninteractive + for i in $(seq 1 30); do getent hosts deb.debian.org >/dev/null && break; sleep 1; done + apt-get update -qq + apt-get -y -qq full-upgrade + apt-get install -y -qq --no-install-recommends ca-certificates curl gnupg sudo kmod jq less rsync locales + EOF + # rsync: scripts/deploy-stack.sh pushes stacks with it. + when: "! pct exec {{ ctid }} -- sh -c 'command -v sudo && command -v jq && command -v rsync && command -v locale-gen' >/dev/null 2>&1" + + - name: Generate the en_US.UTF-8 locale the template's LANG names + # The Debian template sets LANG=en_US.UTF-8 but ships no locale, so every + # apt and perl run warns about it. + shell: | + pct exec {{ ctid }} -- bash -s <<'EOF' + set -euo pipefail + sed -i 's/^# *en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen + locale-gen >/dev/null + EOF + when: "! pct exec {{ ctid }} -- locale -a 2>/dev/null | grep -qi '^en_US.utf8$'" + + - name: Fleet identities (docker 851, infra-ops 850, vh 1000) + /opt/docker tree + shell: | + pct exec {{ ctid }} -- bash -s <<'EOF' + set -euo pipefail + getent group docker >/dev/null || groupadd -g 851 docker + getent group infra-ops >/dev/null || groupadd -g 850 infra-ops + id infra-ops >/dev/null 2>&1 || useradd -u 850 -g 850 -G docker -m -s /bin/bash infra-ops + chmod 0700 /home/infra-ops + install -d -m 0700 -o infra-ops -g infra-ops /home/infra-ops/.ssh + echo '{{ infra_ops_pubkey }}' > /home/infra-ops/.ssh/authorized_keys + chown infra-ops:infra-ops /home/infra-ops/.ssh/authorized_keys + chmod 0600 /home/infra-ops/.ssh/authorized_keys + echo 'infra-ops ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/infra-ops + chmod 0440 /etc/sudoers.d/infra-ops + id vh >/dev/null 2>&1 || useradd -u 1000 -U -G docker,sudo -m -s /bin/bash vh + chmod 0700 /home/vh + install -d -m 2775 -o root -g docker /opt/docker /opt/docker/compose /opt/docker/conf + EOF + when: "! pct exec {{ ctid }} -- sh -c 'test \"$(id -u infra-ops)\" = 850 && test \"$(getent group docker | cut -d: -f3)\" = 851 && test -d /opt/docker/compose'" + + - name: Hugging Face cache for the embed-rerank stack + shell: pct exec {{ ctid }} -- install -d -m 2775 -o root -g docker /opt/aimodels /opt/aimodels/huggingface + when: "! pct exec {{ ctid }} -- test -d /opt/aimodels/huggingface" + + - name: NVIDIA userspace {{ driver_version }} from the host's .run (no kernel modules) + shell: | + set -e + f=NVIDIA-Linux-x86_64-{{ driver_version }}-no-compat32.run + pct push {{ ctid }} {{ stage_dir }}/$f /root/$f --perms 0755 + pct exec {{ ctid }} -- sh /root/$f --no-kernel-modules --silent --no-x-check --no-opengl-files + pct exec {{ ctid }} -- rm -f /root/$f + when: "! pct exec {{ ctid }} -- sh -c 'nvidia-smi --query-gpu=driver_version --format=csv,noheader 2>/dev/null' | grep -qx '{{ driver_version }}'" + + - name: docker-ce from Docker's apt repo + shell: | + pct exec {{ ctid }} -- bash -s <<'EOF' + set -euo pipefail + export DEBIAN_FRONTEND=noninteractive + install -m 0755 -d /etc/apt/keyrings + curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc + chmod a+r /etc/apt/keyrings/docker.asc + echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian bookworm stable" \ + > /etc/apt/sources.list.d/docker.list + apt-get update -qq + apt-get install -y -qq docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin + EOF + when: "! pct exec {{ ctid }} -- sh -c 'command -v docker' >/dev/null 2>&1" + + - name: nvidia-container-toolkit, wired into dockerd, no-cgroups for the LXC + shell: | + pct exec {{ ctid }} -- bash -s <<'EOF' + set -euo pipefail + export DEBIAN_FRONTEND=noninteractive + curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \ + | gpg --dearmor --yes -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg + curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \ + | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \ + > /etc/apt/sources.list.d/nvidia-container-toolkit.list + apt-get update -qq + apt-get install -y -qq nvidia-container-toolkit + nvidia-ctk runtime configure --runtime=docker + nvidia-ctk config --set nvidia-container-cli.no-cgroups=true --in-place + systemctl restart docker + EOF + when: "! pct exec {{ ctid }} -- sh -c 'grep -q nvidia /etc/docker/daemon.json && grep -Eq \"^no-cgroups *= *true\" /etc/nvidia-container-runtime/config.toml' 2>/dev/null" + +verify: + - name: Container is running with onboot set, started after the core guests + shell: "pct status {{ ctid }} | grep -q running && pct config {{ ctid }} | grep -q '^onboot: 1' && pct config {{ ctid }} | grep -q '^startup: order=30'" + changed_when: "false" + + - name: nvidia-smi inside the CT reports the host's driver version + shell: pct exec {{ ctid }} -- nvidia-smi --query-gpu=name,driver_version --format=csv,noheader | grep -q 'RTX 2000.*, {{ driver_version }}$' + changed_when: "false" + + - name: A Docker container gets the GPU (toolkit + no-cgroups work end to end) + shell: pct exec {{ ctid }} -- docker run --rm --gpus all debian:12-slim nvidia-smi -L | grep -q 'RTX 2000' + changed_when: "false" + + - name: Fleet identities are the pinned ids + shell: | + pct exec {{ ctid }} -- sh -c 'test "$(id -u infra-ops)" = 850 && test "$(id -u vh)" = 1000 && test "$(getent group docker | cut -d: -f3)" = 851 && id -nG infra-ops | grep -qw docker' + changed_when: "false" diff --git a/playbooks/esh-pve-nvidia-host.yaml b/playbooks/esh-pve-nvidia-host.yaml new file mode 100644 index 0000000..8a218c9 --- /dev/null +++ b/playbooks/esh-pve-nvidia-host.yaml @@ -0,0 +1,137 @@ +# esh-pve — NVIDIA driver on the HYPERVISOR, for the RTX 2000 Ada that an LXC +# (esh-ml1, embedding + reranking) uses. +# +# WHY A HOST DRIVER: Prime's decision 2026-09-24 — LXC + host driver, NOT a +# VFIO VM. Both esh-pve hangs this year (the 2026-08-19 hard freeze and the +# 2026-09-10 vzdump that left `kvm -id 102` in D state for 14 days) came from +# VFIO passthrough on this box. An LXC shares the host kernel's driver, so no +# passthrough, no pinned VM RAM. Accepted cost: a driver on the hypervisor, +# rebuilt by DKMS on every PVE kernel update, and the LXC's userspace libraries +# must be the SAME version as this kernel module (see +# playbooks/esh-ml1-lxc.yaml, which installs from the same .run file). +# +# WHY THE .run FILE, NOT AN APT REPO: the LXC must match the host module +# exactly. One file, installed with the kernel module here and with +# --no-kernel-module in the container, makes the match true by construction. +# An apt repo on both sides would let a routine `apt upgrade` move one side +# and not the other, which breaks every CUDA call in the container. +# +# WHY 580: the fleet's vLLM image (vllm/vllm-openai:v0.24.0) is a CUDA 13.0 +# build and needs a >= 580 driver. fv-ml1 runs 580.65.06; this is the newest +# point release on the same branch. `-no-compat32` because a headless compute +# box has no use for 32-bit libraries. +# +# NO REBOOT NEEDED: nouveau is blacklisted and was never loaded, and the T400's +# vfio-pci ids no longer match any device, so nothing holds the card. The +# module loads live. (esh-pve is ESH's only DNS and mesh route; a reboot takes +# the whole site dark from outside — avoid one.) +# +# Upgrading the driver: bump driver_version + driver_sha256 here AND in +# playbooks/esh-ml1-lxc.yaml, run this, then that. Until both have run, CUDA +# in the container fails with "driver/library version mismatch". +# +# Run: scripts/elway root@esh-pve --playbook playbooks/esh-pve-nvidia-host.yaml + +vars: + driver_version: 580.178.04 + # From download.nvidia.com/XFree86/Linux-x86_64/580.178.04/*.run.sha256sum + driver_sha256: 1833c8c5c53481dfab46df442c3de26ec5d4c35084e03d94ad93746d29899750 + stage_dir: /root/nvidia + +steps: + - name: Stage the driver .run and check it against NVIDIA's published sha256 + shell: | + set -e + mkdir -p {{ stage_dir }} + f={{ stage_dir }}/NVIDIA-Linux-x86_64-{{ driver_version }}-no-compat32.run + if ! echo "{{ driver_sha256 }} $f" | sha256sum -c --status 2>/dev/null; then + curl -fsSL -o "$f.part" \ + https://download.nvidia.com/XFree86/Linux-x86_64/{{ driver_version }}/NVIDIA-Linux-x86_64-{{ driver_version }}-no-compat32.run + echo "{{ driver_sha256 }} $f.part" | sha256sum -c + mv "$f.part" "$f" + chmod 0755 "$f" + fi + when: "! echo '{{ driver_sha256 }} {{ stage_dir }}/NVIDIA-Linux-x86_64-{{ driver_version }}-no-compat32.run' | sha256sum -c --status 2>/dev/null" + + # The T400 was passed to VM 102 through vfio-pci by id. The card and the VM + # are both gone; the ids match nothing. Kept (not deleted) so the retirement + # is readable later. + - name: Retire the T400's vfio-pci id binding + shell: mv /etc/modprobe.d/vfio.conf {{ stage_dir }}/vfio.conf.retired-2026-09-24 + when: "grep -q '10de:1ff2' /etc/modprobe.d/vfio.conf 2>/dev/null" + + # blacklist.conf blacklisted `nvidia` alongside nouveau (VM-passthrough era, + # when the host must never touch the card). nouveau stays blacklisted. + - name: Stop blacklisting the nvidia module (nouveau stays blacklisted) + shell: | + cp -n /etc/modprobe.d/blacklist.conf {{ stage_dir }}/blacklist.conf.pre-nvidia + sed -i '/^blacklist nvidia$/d' /etc/modprobe.d/blacklist.conf + when: "grep -qx 'blacklist nvidia' /etc/modprobe.d/blacklist.conf" + + - name: Install the driver with DKMS (open kernel modules) + # --no-opengl-files: headless compute. --kernel-module-type=open: Ada + # (Turing+) is supported by the open modules and fv-ml1 runs them too. + # DKMS rebuilds the module whenever a new proxmox-headers package lands; + # `proxmox-headers-6.8` (the meta-package) is installed so new headers + # arrive with each new kernel. + shell: | + sh {{ stage_dir }}/NVIDIA-Linux-x86_64-{{ driver_version }}-no-compat32.run \ + --dkms --silent --no-x-check --no-opengl-files --kernel-module-type=open \ + --kernel-name="$(uname -r)" + when: "! dkms status -m nvidia -v {{ driver_version }} -k $(uname -r) 2>/dev/null | grep -q installed" + + - name: Load nvidia + nvidia_uvm at every boot + shell: | + printf '# RTX 2000 Ada for the esh-ml1 LXC — see playbooks/esh-pve-nvidia-host.yaml\nnvidia\nnvidia_uvm\n' \ + > /etc/modules-load.d/nvidia.conf + when: "! grep -qx 'nvidia_uvm' /etc/modules-load.d/nvidia.conf 2>/dev/null" + + # The LXC binds /dev/nvidia0, /dev/nvidiactl and /dev/nvidia-uvm{,-tools}. + # Those nodes do not exist until something asks for them, and `pct start` + # refuses a dev passthrough whose host path is missing. This unit creates + # them and keeps the GPU initialised (persistence mode), and runs BEFORE + # pve-guests so onboot guests find them. If it fails, only the GPU LXC fails + # to start; the other guests do not depend on it. + - name: Install the GPU-init + persistence unit, ordered before pve-guests + shell: | + cat > /etc/systemd/system/nvidia-persistenced.service <<'EOF' + # See playbooks/esh-pve-nvidia-host.yaml + [Unit] + Description=NVIDIA persistence daemon + device nodes for the GPU LXC + Before=pve-guests.service + After=systemd-modules-load.service + + [Service] + Type=forking + # -c0 -u: load nvidia_uvm and create /dev/nvidia-uvm{,-tools} and /dev/nvidia0. + ExecStartPre=/usr/bin/nvidia-modprobe -c0 -u + ExecStart=/usr/bin/nvidia-persistenced --user root + ExecStopPost=/bin/rm -rf /var/run/nvidia-persistenced + + [Install] + WantedBy=multi-user.target + EOF + systemctl daemon-reload + systemctl enable --now nvidia-persistenced.service + when: "! systemctl is-active --quiet nvidia-persistenced.service || ! grep -q 'Before=pve-guests.service' /etc/systemd/system/nvidia-persistenced.service" + +verify: + - name: nvidia-smi sees the RTX 2000 Ada at the pinned driver version + shell: nvidia-smi --query-gpu=name,driver_version --format=csv,noheader | grep -q 'RTX 2000.*, {{ driver_version }}$' + changed_when: "false" + + - name: Kernel module in use is the DKMS-built {{ driver_version }} + shell: grep -q '{{ driver_version }}' /proc/driver/nvidia/version && dkms status -m nvidia -v {{ driver_version }} -k $(uname -r) | grep -q installed + changed_when: "false" + + - name: Device nodes the LXC binds all exist + shell: test -c /dev/nvidia0 && test -c /dev/nvidiactl && test -c /dev/nvidia-uvm && test -c /dev/nvidia-uvm-tools + changed_when: "false" + + - name: Persistence mode is on + shell: nvidia-smi --query-gpu=persistence_mode --format=csv,noheader | grep -qx Enabled + changed_when: "false" + + - name: Survives a reboot (modules-load + unit enabled before pve-guests) + shell: grep -qx nvidia_uvm /etc/modules-load.d/nvidia.conf && systemctl is-enabled --quiet nvidia-persistenced.service && ! grep -qx 'blacklist nvidia' /etc/modprobe.d/blacklist.conf + changed_when: "false" diff --git a/servers/esh-docker-vm/README.md b/servers/esh-docker-vm/README.md index dd5f67b..3ff2b64 100644 --- a/servers/esh-docker-vm/README.md +++ b/servers/esh-docker-vm/README.md @@ -73,3 +73,32 @@ Like `nh3-docker`, this host runs **Dozzle** and **Beszel** agents that report b ## Placement rule Home-lab workloads for the ESH site go here. Not part of the PFI colo topology. + +## ⚠ Recurring: `homepage` wedges unkillably (3× — 2026-06-03, 2026-09-18, 2026-09-24) + +The Homepage container stops answering (Uptime Kuma: `timeout of 16000ms +exceeded`; healthcheck `Connecting to 127.0.0.1:3000` times out) and cannot be +stopped. Signature captured 2026-09-24 ~2224 PT, guest kernel `6.1.0-41-amd64`: + +- one `node` thread in **D state in `vm_mmap_pgoff`** — waiting for its own + process's `mmap_lock` for write, with **no visible holder** (every other + thread sat in `futex_wait`; a scan of every task's kernel stack found no + reader in a fault, NFS or `access_remote_vm` path except the `ps` calls + queued behind it). +- `docker restart` → *"tried to kill container, but did not receive an exit + event"*; the process then sits in **`exit_mmap`** (uninterruptible) with + PID 1 of the container in `zap_pid_ns_processes`. The June entry in + archival-memory records the same `exit_mmap` end state. +- The rest of the VM is fine: DNS (AdGuard) answered, 17 other containers up. + +⚠ **Diagnosing it can hang your shell.** `ps`, `pgrep` and `docker top` read +`/proc//cmdline|environ`, which takes the same lock, so they block in D +state too. Read `/proc//task/*/stat` and `sudo cat .../stack` instead +(`ssh infra-ops@10.0.50.45`, NOPASSWD). + +**Only a VM reboot clears it**, and this VM is ESH's DNS resolver, so the +reboot is a short ESH-wide DNS outage — schedule it. Root cause not +established; a kernel-side mmap_lock problem is the leading guess, not a +finding. Moving Homepage to ana-docker (as was done for Uptime Kuma on +2026-09-21 for the same box's history) would take the dashboard out of this +failure domain. diff --git a/servers/esh-ml1/README.md b/servers/esh-ml1/README.md new file mode 100644 index 0000000..0a55fb6 --- /dev/null +++ b/servers/esh-ml1/README.md @@ -0,0 +1,83 @@ +# esh-ml1 + +GPU LXC for the ESH home lab: **CT 110 on esh-pve**, holding the **NVIDIA RTX +2000E Ada** (16 GB, 50 W, `01:00.0`). It serves the fleet's embedding and +reranking models locally at ESH. Built 2026-09-24. + +| | | +|---|---| +| **IP** | `10.0.50.80/24`, VLAN 50, gateway `10.0.50.1` (static, outside the UDM's `.150–.250` DHCP pool) | +| **DNS** | `esh-ml1.esh.internal` | +| **SSH** | `ssh esh-ml1` → `infra-ops@10.0.50.80` (NOPASSWD sudo) · from the host: `pct enter 110` | +| **OS** | Debian 12, unprivileged, `nesting=1,keyctl=1` | +| **Size** | 6 cores, 16 GB RAM + 2 GB swap, 80 GB rootfs on `local-lvm` | +| **Boot** | `onboot: 1`, `startup: order=30` — after esh-scale (1), esh-vm-db (10) and esh-vm-docker (20), so a GPU fault never delays ESH's DNS or mesh route | +| **Backups** | **None, on purpose.** esh-pve's vzdump job lists vmids explicitly and 110 is not one. Everything is rebuilt from the playbooks and the stack; models re-download. | + +## What it serves + +`stacks/embed-rerank` (`/opt/docker/compose/embed-rerank`): + +| container | model | port | gateway name | +|---|---|---|---| +| `vllm-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 | `qwen3-embedding` (order 2) | +| `vllm-rerank-bge` | `BAAI/bge-reranker-v2-m3` | 8013 | `reranker` (order 2) | + +The same models, vLLM version (`v0.24.0`, digest `251eba5cc7c1`) and flags as +fv-ml1's `vllm` stack, so the two sites are interchangeable. In LiteLLM they +are the **order-2 failover** behind fv-ml1: fv-ml1 serves every request while +it is up. + +**Parity, measured 2026-09-24** (11 texts incl. CJK, code, a 6k-char passage; +2 runs per site): + +| | median | min | +|---|---|---| +| embed cosine FV vs ESH, same text | 0.999908 | 0.999772 | +| noise floor FV vs FV | 0.999927 | 0.999791 | +| noise floor ESH vs ESH | 0.999911 | 0.999809 | +| negative control, different texts | 0.232 | 0.071 | + +The cross-site difference is inside each site's own run-to-run noise; this +method cannot resolve a cosine gap below ~2×10⁻⁴. Reranker scores differed by +at most 0.000145 (FV-vs-FV floor 0.000181), with identical ranking. + +**VRAM:** 0.20 × 16,380 MiB each; 4,823 MiB in use with both loaded, ~11 GB +free. + +## How it is built + +1. [`playbooks/esh-pve-nvidia-host.yaml`](../../playbooks/esh-pve-nvidia-host.yaml) + — driver **580.178.04** (open modules, DKMS) on the **hypervisor**, the + `nvidia-persistenced` unit that creates the device nodes before + `pve-guests`, and removal of the old VFIO/blacklist config. +2. [`playbooks/esh-ml1-lxc.yaml`](../../playbooks/esh-ml1-lxc.yaml) — the CT, + `dev0–3` GPU nodes, the NVIDIA userspace from the **same `.run`** with + `--no-kernel-modules`, fleet ids (infra-ops 850, docker 851, vh 1000), + docker-ce and nvidia-container-toolkit with `no-cgroups = true`. +3. `scripts/deploy-stack.sh esh-ml1 embed-rerank`, then `docker compose up -d`. + +Both playbooks are idempotent; re-run them to repair. + +## ⚠ Driver version lock + +The kernel module lives on esh-pve; the libraries live in this container. They +**must be the same version**, or every CUDA call fails with *"driver/library +version mismatch"*. To upgrade: bump `driver_version` + `driver_sha256` in the +host playbook and `driver_version` in the LXC playbook, run the host one, then +the LXC one, then restart the stack. + +A PVE kernel update is handled by DKMS (`proxmox-headers-6.8` pulls headers +for each new kernel). Moving esh-pve to a different kernel series (6.14 opt-in) +needs that series' headers meta-package installed first, or the module will not +build and this CT will fail to start at the next boot. + +## Not yet wired + +- **Homepage**: the compose carries labels, but esh-ml1 is not in + `stacks/homepage/conf/docker.yaml` (it would need dockerd on tcp/2375 like + the other hosts). +- **Beszel**: no agent yet. +- **ESH consumers** (Open WebUI RAG, Paperless) still go through the gateway + at ana-docker, so they do not survive a mesh outage. See persistent-memory + for the open decision. diff --git a/servers/esh-ml1/ssh-target b/servers/esh-ml1/ssh-target new file mode 100644 index 0000000..b2b66e5 --- /dev/null +++ b/servers/esh-ml1/ssh-target @@ -0,0 +1 @@ +infra-ops@10.0.50.80 diff --git a/servers/esh-ml1/system-details.txt b/servers/esh-ml1/system-details.txt new file mode 100644 index 0000000..0dd33b3 --- /dev/null +++ b/servers/esh-ml1/system-details.txt @@ -0,0 +1,267 @@ + +===== HOST ===== + +Hostname: esh-ml1.esteban.net +Date: 2026-09-25T05:35:31+00:00 +Uptime: up 20 minutes +OS: Debian GNU/Linux 12 (bookworm) +Kernel: 6.8.12-42-pve +Arch: x86_64 + +===== HARDWARE ===== + +CPU cores: 6 +CPU model: 13th Gen Intel(R) Core(TM) i9-13900H +MemTotal: 16.0 GB +MemAvailable: 9.8 GB + +===== GPUS ===== + +index, name, memory.total [MiB], memory.free [MiB], driver_version +0, NVIDIA RTX 2000E Ada Generation, 16380 MiB, 11072 MiB, 580.178.04 + +===== FILESYSTEMS (df) ===== + +Filesystem Size Used Avail Use% Mounted on +/dev/mapper/pve-vm--110--disk--0 79G 34G 41G 45% / + +===== PERSISTENT MOUNTS (/etc/fstab, non-comment) ===== + + +===== TARGETED DATA PATHS ===== + +/opt (total: 3.3G) + total 20 + drwxr-xr-x 5 root root 4096 2026-09-25 05:16 . + drwxr-xr-x 18 root root 4096 2026-09-25 05:14 .. + drwxrwsr-x 3 root docker 4096 2026-09-25 05:16 aimodels + drwx--x--x 4 root root 4096 2026-09-25 05:15 containerd + drwxrwsr-x 4 root docker 4096 2026-09-25 05:15 docker + +/opt/docker (total: 32K) + total 16 + drwxrwsr-x 4 root docker 4096 2026-09-25 05:15 . + drwxr-xr-x 5 root root 4096 2026-09-25 05:16 .. + drwxrwsr-x 3 root docker 4096 2026-09-25 05:16 compose + drwxrwsr-x 2 root docker 4096 2026-09-25 05:15 conf + +/opt/docker/compose (total: 24K) + total 12 + drwxrwsr-x 3 root docker 4096 2026-09-25 05:16 . + drwxrwsr-x 4 root docker 4096 2026-09-25 05:15 .. + drwxrwsr-x 2 root docker 4096 2026-09-25 05:17 embed-rerank + +/opt/docker/conf (total: 4.0K) + total 8 + drwxrwsr-x 2 root docker 4096 2026-09-25 05:15 . + drwxrwsr-x 4 root docker 4096 2026-09-25 05:15 .. + +/var/lib/docker (total: 4.0K) + +/srv (total: 4.0K) + total 8 + drwxr-xr-x 2 root root 4096 2025-09-07 15:14 . + drwxr-xr-x 18 root root 4096 2026-09-25 05:14 .. + + +===== DOCKER ===== + +Server: 29.8.1 Client: 29.8.1 + +----- docker info ----- +Containers: 2 (running 2, paused 0, stopped 0) +Images: 2 +Runtimes: map[io.containerd.runc.v2:{{runc [] map[]} map[org.opencontainers.runtime-spec.features:{"ociVersionMin":"1.0.0","ociVersionMax":"1.3.0","hooks":["prestart","createRuntime","createContainer","startContainer","poststart","poststop"],"mountOptions":["async","atime","bind","defaults","dev","diratime","dirsync","exec","iversion","lazytime","loud","mand","noatime","nodev","nodiratime","noexec","noiversion","nolazytime","nomand","norelatime","nostrictatime","nosuid","nosymfollow","private","ratime","rbind","rdev","rdiratime","relatime","remount","rexec","rnoatime","rnodev","rnodiratime","rnoexec","rnorelatime","rnostrictatime","rnosuid","rnosymfollow","ro","rprivate","rrelatime","rro","rrw","rshared","rslave","rstrictatime","rsuid","rsymfollow","runbindable","rw","shared","silent","slave","strictatime","suid","symfollow","sync","tmpcopyup","unbindable"],"linux":{"namespaces":["cgroup","ipc","mount","network","pid","time","user","uts"],"capabilities":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_DAC_READ_SEARCH","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_SETGID","CAP_SETUID","CAP_SETPCAP","CAP_LINUX_IMMUTABLE","CAP_NET_BIND_SERVICE","CAP_NET_BROADCAST","CAP_NET_ADMIN","CAP_NET_RAW","CAP_IPC_LOCK","CAP_IPC_OWNER","CAP_SYS_MODULE","CAP_SYS_RAWIO","CAP_SYS_CHROOT","CAP_SYS_PTRACE","CAP_SYS_PACCT","CAP_SYS_ADMIN","CAP_SYS_BOOT","CAP_SYS_NICE","CAP_SYS_RESOURCE","CAP_SYS_TIME","CAP_SYS_TTY_CONFIG","CAP_MKNOD","CAP_LEASE","CAP_AUDIT_WRITE","CAP_AUDIT_CONTROL","CAP_SETFCAP","CAP_MAC_OVERRIDE","CAP_MAC_ADMIN","CAP_SYSLOG","CAP_WAKE_ALARM","CAP_BLOCK_SUSPEND","CAP_AUDIT_READ","CAP_PERFMON","CAP_BPF","CAP_CHECKPOINT_RESTORE"],"cgroup":{"v1":true,"v2":true,"systemd":true,"systemdUser":true,"rdma":true},"seccomp":{"enabled":true,"actions":["SCMP_ACT_ALLOW","SCMP_ACT_ERRNO","SCMP_ACT_KILL","SCMP_ACT_KILL_PROCESS","SCMP_ACT_KILL_THREAD","SCMP_ACT_LOG","SCMP_ACT_NOTIFY","SCMP_ACT_TRACE","SCMP_ACT_TRAP"],"operators":["SCMP_CMP_EQ","SCMP_CMP_GE","SCMP_CMP_GT","SCMP_CMP_LE","SCMP_CMP_LT","SCMP_CMP_MASKED_EQ","SCMP_CMP_NE"],"archs":["SCMP_ARCH_AARCH64","SCMP_ARCH_ARM","SCMP_ARCH_LOONGARCH64","SCMP_ARCH_MIPS","SCMP_ARCH_MIPS64","SCMP_ARCH_MIPS64N32","SCMP_ARCH_MIPSEL","SCMP_ARCH_MIPSEL64","SCMP_ARCH_MIPSEL64N32","SCMP_ARCH_PPC","SCMP_ARCH_PPC64","SCMP_ARCH_PPC64LE","SCMP_ARCH_RISCV64","SCMP_ARCH_S390","SCMP_ARCH_S390X","SCMP_ARCH_X32","SCMP_ARCH_X86","SCMP_ARCH_X86_64"],"knownFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG","SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV"],"supportedFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"]},"apparmor":{"enabled":true},"selinux":{"enabled":true},"intelRdt":{"enabled":true,"schemata":true,"monitoring":true},"memoryPolicy":{"modes":["MPOL_BIND","MPOL_DEFAULT","MPOL_INTERLEAVE","MPOL_LOCAL","MPOL_PREFERRED","MPOL_PREFERRED_MANY","MPOL_WEIGHTED_INTERLEAVE"],"flags":["MPOL_F_NUMA_BALANCING","MPOL_F_RELATIVE_NODES","MPOL_F_STATIC_NODES"]},"mountExtensions":{"idmap":{"enabled":true}},"netDevices":{"enabled":true}},"annotations":{"io.github.seccomp.libseccomp.version":"2.5.4","org.opencontainers.runc.checkpoint.enabled":"true","org.opencontainers.runc.commit":"v1.5.1-0-g8f2685a4","org.opencontainers.runc.version":"1.5.1"},"potentiallyUnsafeConfigAnnotations":["bundle","org.systemd.property.","org.criu.config"]}]} nvidia:{{nvidia-container-runtime [] map[]} map[org.opencontainers.runtime-spec.features:{"ociVersionMin":"1.0.0","ociVersionMax":"1.3.0","hooks":["prestart","createRuntime","createContainer","startContainer","poststart","poststop"],"mountOptions":["async","atime","bind","defaults","dev","diratime","dirsync","exec","iversion","lazytime","loud","mand","noatime","nodev","nodiratime","noexec","noiversion","nolazytime","nomand","norelatime","nostrictatime","nosuid","nosymfollow","private","ratime","rbind","rdev","rdiratime","relatime","remount","rexec","rnoatime","rnodev","rnodiratime","rnoexec","rnorelatime","rnostrictatime","rnosuid","rnosymfollow","ro","rprivate","rrelatime","rro","rrw","rshared","rslave","rstrictatime","rsuid","rsymfollow","runbindable","rw","shared","silent","slave","strictatime","suid","symfollow","sync","tmpcopyup","unbindable"],"linux":{"namespaces":["cgroup","ipc","mount","network","pid","time","user","uts"],"capabilities":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_DAC_READ_SEARCH","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_SETGID","CAP_SETUID","CAP_SETPCAP","CAP_LINUX_IMMUTABLE","CAP_NET_BIND_SERVICE","CAP_NET_BROADCAST","CAP_NET_ADMIN","CAP_NET_RAW","CAP_IPC_LOCK","CAP_IPC_OWNER","CAP_SYS_MODULE","CAP_SYS_RAWIO","CAP_SYS_CHROOT","CAP_SYS_PTRACE","CAP_SYS_PACCT","CAP_SYS_ADMIN","CAP_SYS_BOOT","CAP_SYS_NICE","CAP_SYS_RESOURCE","CAP_SYS_TIME","CAP_SYS_TTY_CONFIG","CAP_MKNOD","CAP_LEASE","CAP_AUDIT_WRITE","CAP_AUDIT_CONTROL","CAP_SETFCAP","CAP_MAC_OVERRIDE","CAP_MAC_ADMIN","CAP_SYSLOG","CAP_WAKE_ALARM","CAP_BLOCK_SUSPEND","CAP_AUDIT_READ","CAP_PERFMON","CAP_BPF","CAP_CHECKPOINT_RESTORE"],"cgroup":{"v1":true,"v2":true,"systemd":true,"systemdUser":true,"rdma":true},"seccomp":{"enabled":true,"actions":["SCMP_ACT_ALLOW","SCMP_ACT_ERRNO","SCMP_ACT_KILL","SCMP_ACT_KILL_PROCESS","SCMP_ACT_KILL_THREAD","SCMP_ACT_LOG","SCMP_ACT_NOTIFY","SCMP_ACT_TRACE","SCMP_ACT_TRAP"],"operators":["SCMP_CMP_EQ","SCMP_CMP_GE","SCMP_CMP_GT","SCMP_CMP_LE","SCMP_CMP_LT","SCMP_CMP_MASKED_EQ","SCMP_CMP_NE"],"archs":["SCMP_ARCH_AARCH64","SCMP_ARCH_ARM","SCMP_ARCH_LOONGARCH64","SCMP_ARCH_MIPS","SCMP_ARCH_MIPS64","SCMP_ARCH_MIPS64N32","SCMP_ARCH_MIPSEL","SCMP_ARCH_MIPSEL64","SCMP_ARCH_MIPSEL64N32","SCMP_ARCH_PPC","SCMP_ARCH_PPC64","SCMP_ARCH_PPC64LE","SCMP_ARCH_RISCV64","SCMP_ARCH_S390","SCMP_ARCH_S390X","SCMP_ARCH_X32","SCMP_ARCH_X86","SCMP_ARCH_X86_64"],"knownFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG","SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV"],"supportedFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"]},"apparmor":{"enabled":true},"selinux":{"enabled":true},"intelRdt":{"enabled":true,"schemata":true,"monitoring":true},"memoryPolicy":{"modes":["MPOL_BIND","MPOL_DEFAULT","MPOL_INTERLEAVE","MPOL_LOCAL","MPOL_PREFERRED","MPOL_PREFERRED_MANY","MPOL_WEIGHTED_INTERLEAVE"],"flags":["MPOL_F_NUMA_BALANCING","MPOL_F_RELATIVE_NODES","MPOL_F_STATIC_NODES"]},"mountExtensions":{"idmap":{"enabled":true}},"netDevices":{"enabled":true}},"annotations":{"io.github.seccomp.libseccomp.version":"2.5.4","org.opencontainers.runc.checkpoint.enabled":"true","org.opencontainers.runc.commit":"v1.5.1-0-g8f2685a4","org.opencontainers.runc.version":"1.5.1"},"potentiallyUnsafeConfigAnnotations":["bundle","org.systemd.property.","org.criu.config"]}]} runc:{{runc [] map[]} map[org.opencontainers.runtime-spec.features:{"ociVersionMin":"1.0.0","ociVersionMax":"1.3.0","hooks":["prestart","createRuntime","createContainer","startContainer","poststart","poststop"],"mountOptions":["async","atime","bind","defaults","dev","diratime","dirsync","exec","iversion","lazytime","loud","mand","noatime","nodev","nodiratime","noexec","noiversion","nolazytime","nomand","norelatime","nostrictatime","nosuid","nosymfollow","private","ratime","rbind","rdev","rdiratime","relatime","remount","rexec","rnoatime","rnodev","rnodiratime","rnoexec","rnorelatime","rnostrictatime","rnosuid","rnosymfollow","ro","rprivate","rrelatime","rro","rrw","rshared","rslave","rstrictatime","rsuid","rsymfollow","runbindable","rw","shared","silent","slave","strictatime","suid","symfollow","sync","tmpcopyup","unbindable"],"linux":{"namespaces":["cgroup","ipc","mount","network","pid","time","user","uts"],"capabilities":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_DAC_READ_SEARCH","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_SETGID","CAP_SETUID","CAP_SETPCAP","CAP_LINUX_IMMUTABLE","CAP_NET_BIND_SERVICE","CAP_NET_BROADCAST","CAP_NET_ADMIN","CAP_NET_RAW","CAP_IPC_LOCK","CAP_IPC_OWNER","CAP_SYS_MODULE","CAP_SYS_RAWIO","CAP_SYS_CHROOT","CAP_SYS_PTRACE","CAP_SYS_PACCT","CAP_SYS_ADMIN","CAP_SYS_BOOT","CAP_SYS_NICE","CAP_SYS_RESOURCE","CAP_SYS_TIME","CAP_SYS_TTY_CONFIG","CAP_MKNOD","CAP_LEASE","CAP_AUDIT_WRITE","CAP_AUDIT_CONTROL","CAP_SETFCAP","CAP_MAC_OVERRIDE","CAP_MAC_ADMIN","CAP_SYSLOG","CAP_WAKE_ALARM","CAP_BLOCK_SUSPEND","CAP_AUDIT_READ","CAP_PERFMON","CAP_BPF","CAP_CHECKPOINT_RESTORE"],"cgroup":{"v1":true,"v2":true,"systemd":true,"systemdUser":true,"rdma":true},"seccomp":{"enabled":true,"actions":["SCMP_ACT_ALLOW","SCMP_ACT_ERRNO","SCMP_ACT_KILL","SCMP_ACT_KILL_PROCESS","SCMP_ACT_KILL_THREAD","SCMP_ACT_LOG","SCMP_ACT_NOTIFY","SCMP_ACT_TRACE","SCMP_ACT_TRAP"],"operators":["SCMP_CMP_EQ","SCMP_CMP_GE","SCMP_CMP_GT","SCMP_CMP_LE","SCMP_CMP_LT","SCMP_CMP_MASKED_EQ","SCMP_CMP_NE"],"archs":["SCMP_ARCH_AARCH64","SCMP_ARCH_ARM","SCMP_ARCH_LOONGARCH64","SCMP_ARCH_MIPS","SCMP_ARCH_MIPS64","SCMP_ARCH_MIPS64N32","SCMP_ARCH_MIPSEL","SCMP_ARCH_MIPSEL64","SCMP_ARCH_MIPSEL64N32","SCMP_ARCH_PPC","SCMP_ARCH_PPC64","SCMP_ARCH_PPC64LE","SCMP_ARCH_RISCV64","SCMP_ARCH_S390","SCMP_ARCH_S390X","SCMP_ARCH_X32","SCMP_ARCH_X86","SCMP_ARCH_X86_64"],"knownFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG","SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV"],"supportedFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"]},"apparmor":{"enabled":true},"selinux":{"enabled":true},"intelRdt":{"enabled":true,"schemata":true,"monitoring":true},"memoryPolicy":{"modes":["MPOL_BIND","MPOL_DEFAULT","MPOL_INTERLEAVE","MPOL_LOCAL","MPOL_PREFERRED","MPOL_PREFERRED_MANY","MPOL_WEIGHTED_INTERLEAVE"],"flags":["MPOL_F_NUMA_BALANCING","MPOL_F_RELATIVE_NODES","MPOL_F_STATIC_NODES"]},"mountExtensions":{"idmap":{"enabled":true}},"netDevices":{"enabled":true}},"annotations":{"io.github.seccomp.libseccomp.version":"2.5.4","org.opencontainers.runc.checkpoint.enabled":"true","org.opencontainers.runc.commit":"v1.5.1-0-g8f2685a4","org.opencontainers.runc.version":"1.5.1"},"potentiallyUnsafeConfigAnnotations":["bundle","org.systemd.property.","org.criu.config"]}]}] +Default runtime: runc +Storage driver: overlayfs +Root dir: /var/lib/docker +Server version: 29.8.1 + +----- running containers ----- +NAMES IMAGE STATUS PORTS +vllm-embed vllm/vllm-openai:v0.24.0 Up 14 minutes (healthy) 0.0.0.0:8001->8000/tcp, [::]:8001->8000/tcp +vllm-rerank-bge vllm/vllm-openai:v0.24.0 Up 15 minutes (healthy) 0.0.0.0:8013->8000/tcp, [::]:8013->8000/tcp + +----- all containers ----- +NAMES IMAGE STATUS +vllm-embed vllm/vllm-openai:v0.24.0 Up 14 minutes (healthy) +vllm-rerank-bge vllm/vllm-openai:v0.24.0 Up 15 minutes (healthy) + +----- networks ----- +NAME DRIVER SCOPE +bridge bridge local +embed-rerank_default bridge local +host host local +none null local + +----- networks (external, non-default — worth knowing for compose external: true) ----- +embed-rerank_default + +----- named volumes ----- +VOLUME NAME DRIVER + +----- compose projects currently running ----- +embed-rerank + +===== COMPOSE FILES (/opt/docker/compose/) ===== + + +>>> /opt/docker/compose/embed-rerank/compose.yaml +# embed-rerank — the fleet's embedding + reranking models, served locally at ESH +# on esh-ml1 (CT 110 on esh-pve, RTX 2000E Ada, 16 GB). +# +# WHY: Prime's decision 2026-09-24 — the Ada card offloads embedding and +# reranking so ESH consumers (Open WebUI RAG, Paperless) are not tied to one +# seat on fv-ml1. It serves the SAME models as fv-ml1's `vllm` stack, with the +# SAME vLLM version and flags, because embedding vectors are model-specific: +# a different embedding model here would silently poison every index built +# against fv-ml1's. Mirror stacks/vllm/compose.yaml when that changes. +# +# vllm-embed Qwen/Qwen3-Embedding-0.6B → /v1/embeddings :8001 +# vllm-rerank-bge BAAI/bge-reranker-v2-m3 → /rerank, /score :8013 +# +# Ports match fv-ml1 on purpose, so a LiteLLM deployment for either site +# differs only in the host part of api_base. +# +# `vllm-rerank-bge`, not fv-ml1's `vllm-rerank-a3`: that name carries R43 +# bake-off provenance for THAT container; and plain `vllm-rerank` was the +# retired Qwen3-Reranker that measured harmful. Name the model instead. +# +# NO `tnet`/traefik-net: esh-ml1 runs no traefik, and an external network +# that does not exist would stop the stack from starting. Consumers reach the +# published ports directly. +# +# Host setup (driver, LXC, docker, toolkit): playbooks/esh-pve-nvidia-host.yaml +# then playbooks/esh-ml1-lxc.yaml. Tunables live in .env — edit that, not this. + +name: embed-rerank + +services: + vllm-embed: + image: vllm/vllm-openai:${VLLM_VERSION} + container_name: vllm-embed + restart: unless-stopped + ipc: host + ports: + - "${EMBED_PORT}:8000" + volumes: + - /opt/aimodels/huggingface:/hfcache + environment: + - HF_HOME=/hfcache + - HF_HUB_CACHE=/hfcache/hub + - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-} + - VLLM_API_KEY=${API_KEY:-} + command: + - ${EMBED_MODEL} + - --served-model-name + - ${EMBED_MODEL} + - --runner + - pooling + - --host + - 0.0.0.0 + - --port + - "8000" + - --gpu-memory-utilization + - ${EMBED_GPU_MEM_UTIL} + - --max-model-len + - ${EMBED_MAX_MODEL_LEN} + - --dtype + - auto + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: ["0"] + capabilities: [gpu] + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8000/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 180s + labels: + - homepage.group=AI - Eval & Retrieval + - homepage.name=vLLM Embed (esh-ml1) + - homepage.icon=mdi-vector-arrange-below + - homepage.description=Qwen3 Embedding 0.6B via vLLM (esh-ml1, RTX 2000E Ada) + - homepage.href=http://10.0.50.80:${EMBED_PORT}/docs + + vllm-rerank-bge: + image: vllm/vllm-openai:${VLLM_VERSION} + container_name: vllm-rerank-bge + restart: unless-stopped + ipc: host + ports: + - "${RERANK_PORT}:8000" + volumes: + - /opt/aimodels/huggingface:/hfcache + environment: + - HF_HOME=/hfcache + - HF_HUB_CACHE=/hfcache/hub + - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-} + - VLLM_API_KEY=${API_KEY:-} + command: + - ${RERANK_MODEL} + - --served-model-name + - ${RERANK_MODEL} + - --runner + - pooling + # bge-reranker-v2-m3 is natively a cross-encoder; no --hf-overrides. + - --host + - 0.0.0.0 + - --port + - "8000" + - --gpu-memory-utilization + - ${RERANK_GPU_MEM_UTIL} + - --max-model-len + - ${RERANK_MAX_MODEL_LEN} + - --dtype + - auto + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: ["0"] + capabilities: [gpu] + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8000/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 180s + labels: + - homepage.group=AI - Eval & Retrieval + - homepage.name=vLLM Rerank bge-v2-m3 (esh-ml1) + - homepage.icon=mdi-sort-variant + - homepage.description=bge-reranker-v2-m3 via vLLM (esh-ml1, RTX 2000E Ada) + - homepage.href=http://10.0.50.80:${RERANK_PORT}/docs + +===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) ===== + +/opt/docker/conf + +===== LISTENING PORTS ===== + +*:22 +0.0.0.0:8001 +0.0.0.0:8013 +127.0.0.1:25 +[::1]:25 +[::]:8001 +[::]:8013 + +===== MODEL / HUGGINGFACE CACHES ===== + + +===== DOCKER-ADJACENT SYSTEMD SERVICES ===== + +container-getty@1.service running +container-getty@2.service running +containerd.service running +docker.service running + +===== DONE ===== + +Paste the above back into the chat, or pass a path as argv[1] to save. diff --git a/servers/esh-pve/README.md b/servers/esh-pve/README.md index b222f0d..ed5a1c0 100644 --- a/servers/esh-pve/README.md +++ b/servers/esh-pve/README.md @@ -13,7 +13,8 @@ Proxmox VE hypervisor for the ESH home lab (`pve.esteban.net`). Non-PFI scope. - **CPU:** 13th Gen Intel Core i9-13900H - **RAM:** 62.5 GB -- **Kernel:** `6.8.12-16-pve` (Proxmox 8.x) +- **Kernel:** `6.8.12-42-pve`, PVE 8.4.20 (as of 2026-09-24) +- **GPU:** NVIDIA RTX 2000E Ada, 16 GB (`01:00.0`) — driven by the **host** (580.178.04, DKMS), used by the esh-ml1 LXC. See below. - **Storage:** local `pve-root` at 94G (21% used) + two large NFS mounts from `10.0.50.50`: - `/mnt/pve/esh-nas` — 92 TB available on `/mnt/pvestore` - `/mnt/pve/tank-vmbu` — 93 TB, VM backup target (~1.4 TB used) @@ -47,10 +48,39 @@ fully up. esh-vm-docker, esh-vm-db and esh-scale all started by themselves. esh-vm-docker's NFS (`/mnt/books`, `/mnt/backup`) mounted, traefik served with no 403s, and AdGuard resolved. -The new card has **no driver bound**. `/etc/modprobe.d` still lists the T400's IDs -for vfio-pci (`10de:1ff2,10de:10fa`), which no longer match anything; harmless, but -clean it up when the card's use is decided. The intended use is a local -embedding endpoint. +The new card got its driver the same night — see the next section. + +## GPU — host driver + the esh-ml1 LXC (2026-09-24) + +**Prime's decision: LXC + host driver, NOT a VFIO VM.** Both hangs on this box +this year came from VFIO passthrough. The card now runs under the host's own +NVIDIA driver and is shared into **CT 110 `esh-ml1`**, which serves embedding +and reranking ([`servers/esh-ml1/README.md`](../esh-ml1/README.md)). + +- **Driver 580.178.04, open kernel modules, DKMS**, from NVIDIA's `.run` + (`/root/nvidia/`). Applied by + [`playbooks/esh-pve-nvidia-host.yaml`](../../playbooks/esh-pve-nvidia-host.yaml) + **live, with no reboot**: nouveau was never loaded and nothing held the card. +- **`nvidia-persistenced.service`** (ours, in `/etc/systemd/system`) runs + `nvidia-modprobe -c0 -u` and the persistence daemon **before + `pve-guests`**, so `/dev/nvidia0`, `nvidiactl` and `nvidia-uvm{,-tools}` + exist when CT 110 starts. Without them `pct start` refuses the `devN:` + entries. If the unit fails, only CT 110 fails; no other guest depends on it. +- `/etc/modules-load.d/nvidia.conf` loads `nvidia` + `nvidia_uvm`. +- **Retired:** the T400's vfio-pci id binding (`/etc/modprobe.d/vfio.conf`, moved + to `/root/nvidia/vfio.conf.retired-2026-09-24`) and `blacklist nvidia` in + `/etc/modprobe.d/blacklist.conf` (`blacklist nouveau` stays). The `vfio*` + lines in `/etc/modules` were left alone; they bind nothing. The initramfs was + not rebuilt, to keep the boot path untouched; the next kernel update will. +- The installer also enabled NVIDIA's `nvidia-{suspend,hibernate,resume}` + units. They only act on a suspend and are harmless on a server. +- ⚠ **Kernel updates:** DKMS rebuilds the module when a new + `proxmox-headers-6.8.*` arrives (the `proxmox-headers-6.8` meta-package is + installed). Before opting into a different kernel series, install that + series' headers meta-package, or the next boot comes up with no GPU and CT + 110 will not start. +- ⚠ **The host module and CT 110's libraries must stay the same version.** + Upgrade both playbooks together (host first). ## Watchdog — hardware, not software diff --git a/stacks/embed-rerank/.env.example b/stacks/embed-rerank/.env.example new file mode 100644 index 0000000..ca3070b --- /dev/null +++ b/stacks/embed-rerank/.env.example @@ -0,0 +1,29 @@ +# embed-rerank tunables (esh-ml1). Copy to `.env` on the server. +# +# Everything model-shaped here MUST match fv-ml1's stacks/vllm .env: the same +# models, the same vLLM version, the same max-model-len. A drift in the +# embedding model or its version makes this seat's vectors incompatible with +# every index built against fv-ml1's. + +# PINNED to fv-ml1's version. Bump both sites together. +VLLM_VERSION=v0.24.0 + +# Same host ports as fv-ml1 (container listens on 8000). +EMBED_PORT=8001 +RERANK_PORT=8013 + +EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B +RERANK_MODEL=BAAI/bge-reranker-v2-m3 + +# Fractions of the RTX 2000E Ada's 16,380 MiB. fv-ml1 runs 0.03 of a 96 GB +# card (~2.9 GB each); 0.20 here is ~3.2 GB each — the same budget plus a +# little, leaving ~9.5 GB free. +EMBED_GPU_MEM_UTIL=0.20 +RERANK_GPU_MEM_UTIL=0.20 +EMBED_MAX_MODEL_LEN=8192 +RERANK_MAX_MODEL_LEN=8192 + +# fv-ml1's seats run with no API key (LiteLLM fronts them); match that. +API_KEY= +# Both models are public; no token needed. +HF_TOKEN= diff --git a/stacks/embed-rerank/README.md b/stacks/embed-rerank/README.md new file mode 100644 index 0000000..1329869 --- /dev/null +++ b/stacks/embed-rerank/README.md @@ -0,0 +1,37 @@ +# embed-rerank + +The fleet's embedding + reranking models on **esh-ml1** (CT 110 on esh-pve, +RTX 2000E Ada). The second site for `qwen3-embedding` and `reranker`; fv-ml1's +[`vllm`](../vllm/) stack is the first. + +| container | model | port | +|---|---|---| +| `vllm-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 | +| `vllm-rerank-bge` | `BAAI/bge-reranker-v2-m3` | 8013 | + +**Keep it in lockstep with `stacks/vllm`:** same models, same `VLLM_VERSION`, +same `--max-model-len`. The LiteLLM groups treat the two sites as one model, +so any drift in the embedding model or its version silently mixes +incompatible vectors into consumers' indexes. + +## Deploy + +```bash +scripts/deploy-stack.sh esh-ml1 embed-rerank +ssh esh-ml1 'cd /opt/docker/compose/embed-rerank && cp -n .env.example .env && docker compose config -q && docker compose up -d' +``` + +Host prerequisites (driver, LXC, docker, toolkit) are in +[`servers/esh-ml1/README.md`](../../servers/esh-ml1/README.md). + +## Smoke test + +```bash +curl -s http://10.0.50.80:8001/v1/embeddings -H 'content-type: application/json' \ + -d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":"hello"}' | jq '.data[0].embedding | length' # 1024 +curl -s http://10.0.50.80:8013/rerank -H 'content-type: application/json' \ + -d '{"model":"BAAI/bge-reranker-v2-m3","query":"cat","documents":["a cat","a car"]}' | jq '.results[0]' +``` + +Parity against fv-ml1 is recorded in the host README; re-measure after any +version bump on either side. diff --git a/stacks/embed-rerank/compose.yaml b/stacks/embed-rerank/compose.yaml new file mode 100644 index 0000000..29db57a --- /dev/null +++ b/stacks/embed-rerank/compose.yaml @@ -0,0 +1,130 @@ +# embed-rerank — the fleet's embedding + reranking models, served locally at ESH +# on esh-ml1 (CT 110 on esh-pve, RTX 2000E Ada, 16 GB). +# +# WHY: Prime's decision 2026-09-24 — the Ada card offloads embedding and +# reranking so ESH consumers (Open WebUI RAG, Paperless) are not tied to one +# seat on fv-ml1. It serves the SAME models as fv-ml1's `vllm` stack, with the +# SAME vLLM version and flags, because embedding vectors are model-specific: +# a different embedding model here would silently poison every index built +# against fv-ml1's. Mirror stacks/vllm/compose.yaml when that changes. +# +# vllm-embed Qwen/Qwen3-Embedding-0.6B → /v1/embeddings :8001 +# vllm-rerank-bge BAAI/bge-reranker-v2-m3 → /rerank, /score :8013 +# +# Ports match fv-ml1 on purpose, so a LiteLLM deployment for either site +# differs only in the host part of api_base. +# +# `vllm-rerank-bge`, not fv-ml1's `vllm-rerank-a3`: that name carries R43 +# bake-off provenance for THAT container; and plain `vllm-rerank` was the +# retired Qwen3-Reranker that measured harmful. Name the model instead. +# +# NO `tnet`/traefik-net: esh-ml1 runs no traefik, and an external network +# that does not exist would stop the stack from starting. Consumers reach the +# published ports directly. +# +# Host setup (driver, LXC, docker, toolkit): playbooks/esh-pve-nvidia-host.yaml +# then playbooks/esh-ml1-lxc.yaml. Tunables live in .env — edit that, not this. + +name: embed-rerank + +services: + vllm-embed: + image: vllm/vllm-openai:${VLLM_VERSION} + container_name: vllm-embed + restart: unless-stopped + ipc: host + ports: + - "${EMBED_PORT}:8000" + volumes: + - /opt/aimodels/huggingface:/hfcache + environment: + - HF_HOME=/hfcache + - HF_HUB_CACHE=/hfcache/hub + - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-} + - VLLM_API_KEY=${API_KEY:-} + command: + - ${EMBED_MODEL} + - --served-model-name + - ${EMBED_MODEL} + - --runner + - pooling + - --host + - 0.0.0.0 + - --port + - "8000" + - --gpu-memory-utilization + - ${EMBED_GPU_MEM_UTIL} + - --max-model-len + - ${EMBED_MAX_MODEL_LEN} + - --dtype + - auto + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: ["0"] + capabilities: [gpu] + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8000/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 180s + labels: + - homepage.group=AI - Eval & Retrieval + - homepage.name=vLLM Embed (esh-ml1) + - homepage.icon=mdi-vector-arrange-below + - homepage.description=Qwen3 Embedding 0.6B via vLLM (esh-ml1, RTX 2000E Ada) + - homepage.href=http://10.0.50.80:${EMBED_PORT}/docs + + vllm-rerank-bge: + image: vllm/vllm-openai:${VLLM_VERSION} + container_name: vllm-rerank-bge + restart: unless-stopped + ipc: host + ports: + - "${RERANK_PORT}:8000" + volumes: + - /opt/aimodels/huggingface:/hfcache + environment: + - HF_HOME=/hfcache + - HF_HUB_CACHE=/hfcache/hub + - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-} + - VLLM_API_KEY=${API_KEY:-} + command: + - ${RERANK_MODEL} + - --served-model-name + - ${RERANK_MODEL} + - --runner + - pooling + # bge-reranker-v2-m3 is natively a cross-encoder; no --hf-overrides. + - --host + - 0.0.0.0 + - --port + - "8000" + - --gpu-memory-utilization + - ${RERANK_GPU_MEM_UTIL} + - --max-model-len + - ${RERANK_MAX_MODEL_LEN} + - --dtype + - auto + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: ["0"] + capabilities: [gpu] + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8000/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 180s + labels: + - homepage.group=AI - Eval & Retrieval + - homepage.name=vLLM Rerank bge-v2-m3 (esh-ml1) + - homepage.icon=mdi-sort-variant + - homepage.description=bge-reranker-v2-m3 via vLLM (esh-ml1, RTX 2000E Ada) + - homepage.href=http://10.0.50.80:${RERANK_PORT}/docs diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index 11b4e77..c9a1e8d 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -443,11 +443,31 @@ model_list: # explicitly. Operator ruling 2026-08-23. --- # --- Qwen3 embeddings --- + # TWO deployments, one name, ordered: fv-ml1 serves (order 1); esh-ml1 takes + # over only when fv-ml1 fails (order 2 — LiteLLM's order-based fallback, v1.97 + # router.py). Same model, same vLLM version and flags at both sites, so the + # vectors are interchangeable: measured 2026-09-24, cosine FV-vs-ESH median + # 0.999908 (min 0.999772) over 11 texts, inside the FV-vs-FV self-noise floor + # (median 0.999927, min 0.999791); different-text negative control 0.07–0.38. + # ⚠ Failover cost, measured n=3 each: primary REFUSING → +0.15 s; primary + # HOST DOWN (no ARP) → ~18.7 s on EVERY call (no cooldown kicked in). Slow, + # not broken. ⚠ Never add a deployment here that serves a DIFFERENT embedding + # model — vectors are model-specific and a mixed group corrupts indexes. + # esh-ml1: stacks/embed-rerank, playbooks/esh-ml1-lxc.yaml. - model_name: qwen3-embedding litellm_params: model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B api_base: http://10.251.50.54:8001/v1 api_key: os.environ/VLLM_API_KEY + order: 1 + model_info: + mode: embedding + - model_name: qwen3-embedding + litellm_params: + model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B + api_base: http://10.0.50.80:8001/v1 + api_key: os.environ/VLLM_API_KEY + order: 2 model_info: mode: embedding @@ -487,12 +507,24 @@ model_list: enable_thinking: false model_info: mode: chat - # reranker → generic capability name for rerank (currently qwen3-reranker). + # reranker → generic capability name for rerank (bge-reranker-v2-m3 since the + # R43 cutover). fv-ml1 order 1, esh-ml1 order 2 — same failover shape as + # qwen3-embedding above. Parity measured 2026-09-24: |score| FV-vs-ESH max + # 0.000145 vs FV-vs-FV floor 0.000181, identical ranking over 11 documents. - model_name: reranker litellm_params: model: hosted_vllm/BAAI/bge-reranker-v2-m3 api_base: http://10.251.50.54:8013/v1 api_key: os.environ/VLLM_API_KEY + order: 1 + model_info: + mode: rerank + - model_name: reranker + litellm_params: + model: hosted_vllm/BAAI/bge-reranker-v2-m3 + api_base: http://10.0.50.80:8013/v1 + api_key: os.environ/VLLM_API_KEY + order: 2 model_info: mode: rerank