From 5960526c3ff79b4bee24591633abdc25949806d1 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 25 Sep 2026 15:55:33 -0700 Subject: [PATCH] feat(nh3-ml1): second TEI embed/rerank backend live on nh3-pve; parity-verified vs esh-ml1 NH3 site visit done: Secure Boot off, iGPU restored as boot VGA, AMT port cabled. - nh3-pve: NVIDIA 580.178.04 (DKMS, open modules) via pve-nvidia-host.yaml. - nh3-ml1 = CT 109 @ 10.100.50.80 via gpu-lxc.yaml; embed-rerank (TEI 1.9.4) deployed with HOST_NAME/HOST_IP labels. - Parity vs esh-ml1 (1,126 texts, 2 runs/host, controls): embed cosine min 0.999993 = own noise floor; overlap@10 1.000 vs MRL-256 positive control 0.684; rerank top-1 1.00, max diff 0.0014 vs floor 0.0020. On-box speed identical within rep spread. - gpu-lxc.yaml: first step upgrades lxc-pve to >= 6.0.0-2 (Proxmox fix #7006). With 6.0.0-1 every docker run in a nesting CT failed on runc 1.5's sysctl reopen; applied on nh3-pve (one package). - pve-nvidia-host.yaml: document that the headers meta drags in the newest kernel (nh3-pve went 6.8.12-11 -> -43 at the next reboot). - Monitoring: Beszel NVIDIA agent + 5 alerts, Kuma #29/#30, Homepage nh3-ml1-docker, Dozzle agent (hub 8 clients). DNS nh3-ml1.nh3.internal. - nh3-pve README: SB/IGFX/driver/kernel state, btmtk oops on -4x kernels, AMT cabled but unreachable on the network. Gateway routing to nh3-ml1 is not changed. --- CLAUDE.md | 1 + dns/internal.yaml | 3 +- .../2026-09-25-nh3-ml1-live.md | 89 ++++ persistent-memory.md | 85 ++-- playbooks/gpu-lxc.yaml | 19 + playbooks/pve-nvidia-host.yaml | 6 + servers/esh-ml1/README.md | 10 +- servers/nh3-ml1/README.md | 105 ++++ servers/nh3-ml1/ssh-target | 1 + servers/nh3-ml1/system-details.txt | 452 ++++++++++++++++++ servers/nh3-pve/README.md | 100 ++-- stacks/beszel/README.md | 1 + stacks/beszel/hosts/nh3-ml1.yaml | 15 + stacks/dozzle/README.md | 1 + stacks/embed-rerank/README.md | 8 +- stacks/homepage/conf/docker.yaml | 6 + stacks/uptimekuma/monitors.yaml | 25 +- 17 files changed, 815 insertions(+), 112 deletions(-) create mode 100644 persistent-memory.d/2026-09-25-nh3-ml1-live.md create mode 100644 servers/nh3-ml1/README.md create mode 100644 servers/nh3-ml1/ssh-target create mode 100644 servers/nh3-ml1/system-details.txt create mode 100644 stacks/beszel/hosts/nh3-ml1.yaml diff --git a/CLAUDE.md b/CLAUDE.md index 3ab04a2..a51e148 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -367,6 +367,7 @@ Observed and standardized across servers: | esh-pve-nas | 10.0.50.55 | ESH home lab | Proxmox VE hypervisor (storage / media) | `servers/esh-pve-nas/README.md` | | esh-vm-db | 10.0.50.60 | ESH home lab | DB VM — PostgreSQL (paperless-ng) + MongoDB; bare-metal VM, no Docker | `servers/esh-vm-db/README.md` | | esh-ml1 | 10.0.50.80 | ESH home lab | GPU LXC (CT 110 on esh-pve, RTX 2000E Ada 16 GB, host-side NVIDIA driver) — **the fleet's embedding + reranking service** (TEI), sole gateway backend since 2026-09-25 | `servers/esh-ml1/README.md` | +| nh3-ml1 | 10.100.50.80 | NH3 | GPU LXC (CT 109 on nh3-pve, RTX 2000E Ada 16 GB, host-side NVIDIA driver) — second embed/rerank (TEI) backend, twin of esh-ml1; not yet behind the gateway (2026-09-25) | `servers/nh3-ml1/README.md` | **Placement rules:** - GPU-required stacks → `fv-ml1` (primary, Fountain Valley) or `irv-ml1` diff --git a/dns/internal.yaml b/dns/internal.yaml index eab1458..1ebf883 100644 --- a/dns/internal.yaml +++ b/dns/internal.yaml @@ -99,7 +99,7 @@ hosts: - {name: esh-pve, site: esh, v4: 10.0.250.35, note: Proxmox hypervisor} - {name: esh-pve-nas, site: esh, v4: 10.0.50.55, note: Proxmox hypervisor — storage/media} - {name: esh-vm-db, site: esh, v4: 10.0.50.60, note: PostgreSQL + MongoDB} - - {name: esh-ml1, site: esh, v4: 10.0.50.80, note: CT110 on esh-pve — RTX 2000E Ada, embedding + reranking (vLLM)} + - {name: esh-ml1, site: esh, v4: 10.0.50.80, note: CT110 on esh-pve — RTX 2000E Ada, embedding + reranking (TEI) + reward seat} - {name: vm-esh-nas, site: esh, v4: 10.0.50.154, note: NAS-adjacent docker host} - {name: esh-filebot, site: esh, v4: 10.0.50.70, note: restic / file-sync VM} - {name: esh-gw, site: esh, v4: 10.0.250.1, note: esh-gw} @@ -111,6 +111,7 @@ hosts: - {name: ana-scale, site: ana, v4: 10.250.50.45, note: CT114 on pfi-pve — headscale/tailscale subnet router for 10.250.0.0/16} - {name: nh3-headscale, site: nh3, v4: 10.100.50.45, note: CT106 on nh3-pve — headscale control plane (public headscale.phasefinal.com:443)} - {name: nh3-scale, site: nh3, v4: 10.100.50.46, note: CT107 on nh3-pve — subnet router for 10.100.0.0/16} + - {name: nh3-ml1, site: nh3, v4: 10.100.50.80, note: CT109 on nh3-pve — RTX 2000E Ada, second embedding + reranking backend (TEI)} - {name: esh-scale, site: esh, v4: 10.0.50.65, note: CT108 on esh-pve — subnet router for 10.0.0.0/16} # Service aliases — a name that points at whatever host currently runs it, so diff --git a/persistent-memory.d/2026-09-25-nh3-ml1-live.md b/persistent-memory.d/2026-09-25-nh3-ml1-live.md new file mode 100644 index 0000000..9d04a87 --- /dev/null +++ b/persistent-memory.d/2026-09-25-nh3-ml1-live.md @@ -0,0 +1,89 @@ +# `[2026-09-25]` nh3-ml1 LIVE after the NH3 site visit + +**Prime, ~1524 PT:** "secure boot disabled, amt enabled, card plugged." This is the +resume of `2026-09-25-nh3-ml1-standup.md`. nh3-pve came up at 1523. + +## Post-boot checklist: all green + +- Uplink is `enp2s0f1np1`. Guests 100, 101, 102, 105 and CTs 103, 106, 107 are up; + 104 and 108 are off as expected. +- NFS automounts are good on nh3-docker, nh3-dev and pbs-nh3. The post office is + push/reachable. The 14 queued althing messages were all Beszel/Kuma "UP" + recoveries, so nothing needed acting on. +- Secure Boot is disabled. `00:02.0` is back with `boot_vga=1` and i915 loaded. + +## Build (elway, both playbooks idempotent) + +1. `pve-nvidia-host.yaml`: the driver installed (DKMS, 36 s), 6/6 verify. +2. `gpu-lxc.yaml`: CT 109 built, then **verify 3 FAILED**. `docker run` died with + `open sysctl net.ipv4.ip_unprivileged_port_start file: reopen fd 8: permission + denied`. + - The cause is the lxc-pve version, not runc. Both CTs run runc 1.5.1. nh3-pve + (PVE 8.4.1) had lxc-pve **6.0.0-1**, and esh-pve (8.4.20) has **6.0.0-2**, + whose changelog is Proxmox "fix #7006: do not restrict /proc or /sys if + nesting is enabled". + - **Decided autonomously:** upgrade that one package (a dry-run showed + 1 upgraded, 0 new, 0 removed) and `pct reboot 109`. GPU-in-docker then passed. + ops-log recorded. + - The playbook gained a first step that upgrades lxc-pve when it is below + 6.0.0-2, with a quoted name, because an unquoted `#7006` truncated it as a YAML + comment. The re-run was 5/5 verify OK. The guard skips on esh-pve (tested). +3. `embed-rerank` deploy. `deploy-stack.sh --yes` was **denied by the auto-mode + classifier as a blind apply**, so the pattern now is: preview with + `echo n | deploy-stack.sh …` (it shows the diff and aborts), then `echo y |`. + `.env` = example + `HOST_NAME=nh3-ml1`, `HOST_IP=10.100.50.80`. Healthy in about + 120 s including the image pull; 2.7 GB VRAM. + +## Parity and speed (vs esh-ml1) + +Script: session scratchpad, not committed. Corpus: 1,120 repo-doc paragraphs +plus 6 fixed texts, 2 runs per host, 50 queries. +- Embedding cosine esh↔nh3: median 0.999998, min 0.999993. The floors are esh + 0.999995 and nh3 0.999993. +- Negative control (different texts): median 0.50. +- overlap@10 = 1.000 both ways. Positive control, MRL-256: 0.684. +- Rerank, 100×20: top-1 1.00, max score diff 0.0014 (the esh-esh floor is + 0.0020). Positive control, a 4-word query: top-5 order 0.09, max diff 0.97. +- Speed on-box, 3 interleaved reps: short embed p50 6.6–6.7 vs 7.0 ms; bulk + 104–105 vs 106–108 passages/s; rerank-20 p50 156–159 vs 152–161 ms. The same. + +## Monitoring and wiring + +- Beszel: agent (`hosts/nh3-ml1.yaml`, same hub key as esh-ml1), system + `1feeeq61g4mkqre`, 5 alerts, GPU samples present. + - ⚠ zsh gotcha: `for a in "X 1 2"; do set -- $a` does NOT word-split in zsh, so + the first batch of alert POSTs failed silently (empty fields). Use a function + with explicit args. +- Kuma #29/#30 (the seed also refreshed the esh-ml1 descriptions). Homepage + `nh3-ml1-docker` shows 2 cards with no restart. Dozzle agent added, and the hub + `.env` backed up as `.env.bak-20260925-nh3ml1` before the edit. +- DNS `nh3-ml1.nh3.internal` on all 3 resolvers. `~/.ssh/config` `Host nh3-ml1`; + `servers/nh3-ml1/` has ssh-target, system-details and README. + +## Found along the way + +- **Kernel 6.8.12-43 came from our own playbook.** The headers step's + `proxmox-headers-6.8` install upgraded the `proxmox-kernel-6.8` meta + (apt history 14:19:23). The visit reboot booted it. Documented in + `pve-nvidia-host.yaml`. +- **btmtk Bluetooth oops on every -4x boot** (2/2 lines on each of 3 -43 boots, + 0 on the -11 boots; esh-pve -42 is the same). It taints `D`. Only the BT worker + dies. Proposed fix: blacklist btusb on both PVE hosts. Not done. +- **AMT is not reachable.** + - The LM port has a 1 Gb carrier (brief unbridged `ip link up` with IPv6 + disabled, then restored; ops-log recorded). + - No UDM client or lease for `58:47:ca:76:96:0e`. + - A /dev/tcp sweep of 16992/16993 over 10.100.{0,10,50,250}.0/24 found nothing. + Its positive controls, :8390 and :8006, did answer. + - The USW Pro 24's up ports (19, 22, 23, 26) are all other devices. So the cable + is probably on nh3-sw1 (`10.100.250.2`), where SSH times out and no creds are + held. + - Asked Prime. + +## Open (Prime's) + +- Gateway routing for `qwen3-embedding` / `reranker`. Recommendation: load-share + (identical cards, and a second site). +- AMT network settings and switch port. The NanoKVM stays on nh3-pve until AMT KVM + shows the console. +- btusb blacklist on nh3-pve and esh-pve. Low priority. diff --git a/persistent-memory.md b/persistent-memory.md index 55be0c2..cb21978 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-25 ~1430 PT (⭐⭐ nh3-pve SHUT DOWN by Prime for the NH3 site visit — post-boot checklist in Current state. nh3-ml1 blocked on nh3-pve SECURE BOOT; Prime's SB call pending. NanoKVM blind until IGFX fix on the visit; gx10 AC-restore VALIDATED; GPU playbooks host-generic.)_ +_Last updated: 2026-09-25 ~1555 PT (⭐⭐ NH3 visit DONE: SB off, IGFX restored, AMT cabled but NOT on the network. nh3-ml1 LIVE + parity-verified + monitored; gateway routing = Prime's call. lxc-pve fix; kernel -43 btmtk oops noted. gx10 AC-restore VALIDATED.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -115,57 +115,44 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-09-25 ~1430 PT._ +_As of 2026-09-25 ~1555 PT._ -### ⭐⭐ nh3-pve SHUT DOWN 2026-09-25 ~1430 PT (Prime's order) for the NH3 site visit +### ⭐⭐ NH3 site visit DONE; nh3-pve back up 1523; nh3-ml1 LIVE (not yet in the gateway) -This session runs ON nh3-dev and dies with it. nh3-dev wipes /tmp at boot, so there -is NO /tmp handoff after this. **Successor: work this list first.** - -**The site-visit plan.** Prime has mini-DP→DP adapters, so a DisplayPort monitor on -the RTX plus a USB keyboard covers all of it: -1. BIOS: set Primary Display = IGFX. The Auto setting hid the iGPU, so the - NanoKVM is blind. -2. BIOS/Secure Boot: Prime's call is PENDING. (1) SB off (recommended; matches - esh-pve), (2) keep SB and enroll a DKMS MOK via MokManager, or (3) a VFIO VM - instead of the LXC. -3. vPro: cable I226-LM `enp88s0`, provision MEBx, confirm AMT KVM shows the console. -4. Only then move the NanoKVM to pfi-gx10. -Detail and order: `servers/nh3-pve/README.md`. - -**Post-boot checklist:** -1. Network: `ping 10.100.250.60`; `ip -br link`. ⚠ The uplink is now - `enp2s0f1np1` (the cable moved cages on 09-25 at 1116), not `enp2s0f0np0`. Both - are in vmbr0. -2. GPU: `lspci` shows `01:00.0` RTX 2000E. If IGFX was set, `00:02.0` is back with - `boot_vga=1`. `mokutil --sb-state` shows whether Secure Boot is off. -3. Guests running: 100 nh3-docker1, 101 nh3-extdev, 102 nh3-dev, 105 pbs-nh3, - CT 103 nh3-wg, 106 nh3-headscale, 107 nh3-scale. Staying off: 104, 108. -4. NFS automounts: nh3-docker `/mnt/compose` `/mnt/backup`; nh3-dev `/mnt/backup` - `/mnt/smithy`; pbs-nh3 datastore. -5. Post office `postbox status` shows push/reachable; Kuma is green. -6. nh3-dev services: svos :8770, hermes-gateway (Miranda's channel), Booth :8090, - talk, ttyd RO/RW, zellij-web, `dev-backup.timer`. -7. Claude sessions: relaunching them is Prime's call. -8. pfi-gx10: the AC-restore patch is validated, so a power cut brings it back on - its own. -9. Resume nh3-ml1, only once Secure Boot is resolved. - `scripts/elway root@nh3-pve.nh3.internal --playbook playbooks/pve-nvidia-host.yaml`. - Its pre-flight refuses while SB is on without an enrolled key. Then run - `playbooks/gpu-lxc.yaml` with the nh3-ml1 `--var` line from its header, deploy - `embed-rerank` with `.env` `HOST_NAME=nh3-ml1` `HOST_IP=10.100.50.80`, run a - parity check against esh-ml1 (noise floor plus controls), then wire monitoring, - DNS and `servers/nh3-ml1/README.md`. Gateway routing is Prime's call; the - recommendation is load-share after parity. - -**Left on nh3-pve by the failed 09-25 driver install** (harmless, and needed later): -headers 6.8.12-11 plus the series meta, dkms, build-essential; nouveau -blacklisted and unloaded; the `.run` staged in `/root/nvidia`. +Prime, after the visit: "secure boot disabled, amt enabled, card plugged." The +post-boot checklist is all green: the uplink is on `enp2s0f1np1`, all 7 onboot +guests are up, NFS automounts, the post office is push/reachable, and svos, +hermes-gateway, Booth, ttyd, zellij-web and dev-backup are running. +- **Secure Boot OFF. IGFX restored** (`00:02.0` `boot_vga=1`), so the NanoKVM should + have video again (not checked). NVIDIA 580.178.04 DKMS is on the host. +- **nh3-ml1 = CT 109 @ 10.100.50.80, TEI embed+rerank, parity-verified vs esh-ml1: + the two hosts cannot be told apart and run at the same speed.** Monitoring is + wired: Beszel `1feeeq61g4mkqre` with 5 alerts, Kuma #29/#30, Homepage cards, + Dozzle agent (hub 7→8 clients). DNS `nh3-ml1.nh3.internal` is set. + `servers/nh3-ml1/README.md` exists. +- ⏳ **Prime's call: gateway routing.** Recommendation: load-share + `qwen3-embedding` + `reranker` across esh-ml1 and nh3-ml1. +- ⚠ **AMT is cabled (1 Gb link) and reported enabled, but not on the network.** No + UDM lease for `…:96:0e`, and no 16992/16993 on the four NH3 /24s (the sweep was + positive-controlled). It is not on the USW Pro 24, so probably nh3-sw1. Asked + Prime for the MEBx network settings and which switch port. **The NanoKVM stays + on nh3-pve until AMT KVM is confirmed.** +- Found and fixed: **lxc-pve 6.0.0-1 broke Docker in the CT** (runc 1.5 sysctl + reopen denied). Upgraded that one package to 6.0.0-2 (Proxmox fix #7006), and + `gpu-lxc.yaml` now does this itself. +- Found, NOT fixed: the headers install on 09-25 pulled kernel **6.8.12-43**, which + booted at the visit. It oopses in Bluetooth (`btmtk`) on every boot, and so does + esh-pve on -42. Benign so far. The proposed fix is to blacklist `btusb` on both + PVE hosts, effective at the next boot. +- Not yet proven: nh3-ml1 coming back by itself after an nh3-pve reboot. + `nvidia-persistenced` is ordered before `pve-guests`, but only the config was + checked; nothing has been rebooted. +→ `persistent-memory.d/2026-09-25-nh3-ml1-live.md` ### ⭐ esh-ml1 = the fleet's utility GPU host (2026-09-25): embed/rerank (TEI) + reward (vLLM) -TEI is the fleet embed/rerank engine (Prime) and esh-ml1 is the SOLE backend -(⚠ no failover until the second RTX 2000 arrives). The Skywork reward seat moved +TEI is the fleet embed/rerank engine (Prime) and esh-ml1 is the SOLE gateway +backend (nh3-ml1 is live and parity-verified; routing it in is Prime's call). The Skywork reward seat moved there too; the audit found nothing supersedes it. fv-ml1 GPU 1 freed ~15 GB across the day; only `vllm-coder` is left in `stacks/vllm`, and **parakeet stays** on fv-ml1. Monitoring is wired: Beszel with GPU and 5 alerts, Kuma #27/#28, @@ -198,8 +185,7 @@ in `servers/pfi-gx10/README.md`. → `persistent-memory.d/2026-09-24-gx10-ac-res ### Live threads -- **Claude sessions on nh3-dev:** Prime relaunched 3 after the 1116 boot. They die again - with the ~1430 shutdown, and relaunching them is his call. +- **Claude sessions on nh3-dev:** the ~1430 shutdown killed them again; relaunching is Prime's call. - **Homepage `hermes-gateway seat` rename resolved (2026-09-25):** the operator's 2026-09-24 handle split is real (`postbox handles` records it; infra-hermes confirmed the seat daemon carries `hermes-gateway`). The rename was already live and is committed as `422a27c`. - **High Seat backup proof DONE 2026-09-25 0111:** infra-hermes reported and I verified that restic snapshot `cd113bef` (nh3-dev, 0100) contains `~/.high-seat/reports/grok.963ddd3f428b7fc7.jsonl`. Told him to retire the daily 0110 check unless svos-dev asked for an ongoing one. - ESH has a single outside route (esh-scale on esh-pve) — noted, untracked. @@ -207,6 +193,7 @@ in `servers/pfi-gx10/README.md`. → `persistent-memory.d/2026-09-24-gx10-ac-res ## Recent decisions +- `[2026-09-25]` **nh3-ml1 LIVE after the NH3 visit (SB off, IGFX restored): driver + CT 109 + TEI, parity vs esh-ml1 indistinguishable (cos min 0.999993 = own floor; overlap@10 1.000 vs MRL-256 control 0.684), same speed; monitoring/DNS wired. Autonomous: lxc-pve 6.0.0-2 upgrade on nh3-pve (Docker-in-LXC fix #7006). Gateway routing + AMT follow-up are Prime's.** → `persistent-memory.d/2026-09-25-nh3-ml1-live.md` - `[2026-09-25]` **nh3-ml1 = CT 109 @ 10.100.50.80, second TEI embed/rerank backend on nh3-pve's RTX 2000E; GPU playbooks made host-generic — BUILD DEFERRED: blocked on nh3-pve Secure Boot until the NH3 visit; SB handling and gateway routing are Prime's calls.** Tracked: `7ddd116`, Current state. → `persistent-memory.d/2026-09-25-nh3-ml1-standup.md` - `[2026-09-25]` **nh3-pve OOB: HOLD blind until the next NH3 visit (Prime via Miranda, 1333).** Target: NanoKVM → pfi-gx10, and the MS-01 on its own vPro. The IGFX BIOS fix rides the same visit, since AMT KVM captures only the iGPU. Tracked: `servers/nh3-pve/README.md` (`2b49be4`, `cdd7605`). - `[2026-09-25]` **MS-01 foot-gun: a GPU in the PCIe slot renames every NIC** (the slot's root port takes bus 01, so the X710 goes `enp2s0f0np0`→`enp3s0f0np0`). On nh3-pve that means booting with no uplink. Pinned by MAC with `.link` files before the install, and the AMT port was pulled from vmbr0. → `servers/nh3-pve/README.md`, `playbooks/nh3-pve-pin-nic-names.yaml` (`2118449`) diff --git a/playbooks/gpu-lxc.yaml b/playbooks/gpu-lxc.yaml index 5600dc1..072e52e 100644 --- a/playbooks/gpu-lxc.yaml +++ b/playbooks/gpu-lxc.yaml @@ -65,6 +65,25 @@ vars: infra_ops_pubkey: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN+1HBwfXrkfTYWdcnWCjLJ6VLAGC87gxH5h5vKaaA3c infra-ops@pfi-fleet" steps: + # runc >= 1.2.8 (the CVE-2025-52881 fix; the CT gets 1.5.x from docker-ce) + # re-opens /proc/sys files, and lxc-pve 6.0.0-1's AppArmor profile denies it + # in a nesting CT: EVERY `docker run` fails with "open sysctl + # net.ipv4.ip_unprivileged_port_start file: reopen fd 8: permission denied". + # lxc-pve 6.0.0-2 (Proxmox fix #7006) lifts the restriction when nesting is on. + # Measured 2026-09-25: nh3-pve (PVE 8.4.1, lxc-pve 6.0.0-1) failed the GPU + # verify below; esh-pve (8.4.20, 6.0.0-2) never did. One package, no deps, + # running guests unaffected (the change only relaxes the profile). A CT that + # was already running needs a `pct reboot` to pick it up. + - name: "lxc-pve carries fix #7006 (Docker's runc works in a nesting CT)" + shell: | + set -e + export DEBIAN_FRONTEND=noninteractive + apt-get update -qq + apt-get install -y -qq --only-upgrade lxc-pve + v=$(dpkg-query -W -f='${Version}' lxc-pve) + dpkg --compare-versions "$v" ge 6.0.0-2 || { echo "lxc-pve is still $v after upgrade" >&2; exit 1; } + when: "dpkg --compare-versions \"$(dpkg-query -W -f='${Version}' lxc-pve)\" lt 6.0.0-2" + - name: Fetch the Debian 12 template shell: pveam update >/dev/null && pveam download local {{ template }} creates: /var/lib/vz/template/cache/{{ template }} diff --git a/playbooks/pve-nvidia-host.yaml b/playbooks/pve-nvidia-host.yaml index 74193c5..28646bc 100644 --- a/playbooks/pve-nvidia-host.yaml +++ b/playbooks/pve-nvidia-host.yaml @@ -72,6 +72,12 @@ steps: # DKMS needs the headers for the RUNNING kernel (built now) and the series # meta-package (so each future kernel's headers arrive with it and DKMS # rebuilds on upgrade). nh3-pve had neither, nor dkms or a compiler. + # ⚠ The series meta depends on the NEWEST headers, which drags the + # proxmox-kernel meta and the newest kernel image in with it. On nh3-pve + # (2026-09-25) this installed 6.8.12-43 next to the running -11, and the next + # reboot silently booted -43. Expect a kernel change on the next reboot of any + # host that was behind, and before that reboot check that `dkms status` lists + # nvidia for the kernel that will boot, not only for `uname -r`. - name: Kernel headers (running kernel + series meta), dkms, build tools shell: | set -e diff --git a/servers/esh-ml1/README.md b/servers/esh-ml1/README.md index 772510b..5872687 100644 --- a/servers/esh-ml1/README.md +++ b/servers/esh-ml1/README.md @@ -5,10 +5,12 @@ GPU LXC for the ESH home lab: **CT 110 on esh-pve**, holding the **NVIDIA RTX reranking service** — the only backend behind the gateway's `qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3` since 2026-09-25. Built 2026-09-24. -⚠ **Single backend until the second RTX 2000 arrives** (Prime, 2026-09-25). If -esh-ml1, esh-pve, or ESH's mesh route (esh-scale, CT 108) is down, fleet -embeddings and reranking are down: Worldtree recall, nevermore clustering, Open -WebUI RAG. +⚠ **Still the single gateway backend.** The second RTX 2000 is live as +[nh3-ml1](../nh3-ml1/README.md) (2026-09-25, parity-verified: the two hosts cannot +be told apart), but LiteLLM does not route to it yet; that is Prime's call. Until +it does, if esh-ml1, esh-pve, or ESH's mesh route (esh-scale, CT 108) is down, +fleet embeddings and reranking are down: Worldtree recall, nevermore clustering, +Open WebUI RAG. | | | |---|---| diff --git a/servers/nh3-ml1/README.md b/servers/nh3-ml1/README.md new file mode 100644 index 0000000..7c9eeef --- /dev/null +++ b/servers/nh3-ml1/README.md @@ -0,0 +1,105 @@ +# nh3-ml1 + +GPU LXC for the NH3 site: **CT 109 on nh3-pve**, holding the **NVIDIA RTX 2000E +Ada** (16 GB, 50 W, `01:00.0`, PCIe gen4 x8). **It is the second embedding and +reranking backend**, the twin of [esh-ml1](../esh-ml1/README.md): same card, same +driver, same TEI image and models. Built 2026-09-25, after the NH3 site visit +turned Secure Boot off on nh3-pve. + +⚠ **Not behind the gateway yet.** It serves on its own ports and is monitored, but +LiteLLM still routes `qwen3-embedding` and `reranker` to esh-ml1 alone. Routing is +Prime's call (recommendation: load-share; see below). + +| | | +|---|---| +| **IP** | `10.100.50.80/24`, VLAN 50 (`nh3-servers`), gateway `10.100.50.1` (static, outside the UDM's `.150–.249` DHCP pool) | +| **DNS** | `nh3-ml1.nh3.internal` | +| **SSH** | `ssh nh3-ml1` → `infra-ops@10.100.50.80` (NOPASSWD sudo) · from the host: `pct enter 109` | +| **OS** | Debian 12, unprivileged, `nesting=1,keyctl=1` | +| **Size** | 6 cores, 16 GB RAM + 2 GB swap, 80 GB rootfs on `local-zfs` | +| **Boot** | `onboot: 1`, `startup: order=30`, after the site's core guests, so a GPU fault never delays NH3's DNS or mesh route | +| **Backups** | **None, on purpose.** nh3-pve's vzdump job is `all 1`; the playbook added 109 to its `exclude` list. Everything is rebuilt from the playbooks and the stack; models re-download. | + +## What it serves + +`stacks/embed-rerank` (`/opt/docker/compose/embed-rerank`), **TEI 1.9.4** +(`89-1.9.4`). The live `.env` differs from `.env.example` only in +`HOST_NAME=nh3-ml1` and `HOST_IP=10.100.50.80`, which feed the Homepage labels. + +| container | model | port | +|---|---|---| +| `tei-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 (`/v1/embeddings`, `/embed`) | +| `tei-rerank` | `BAAI/bge-reranker-v2-m3` | 8013 (`/rerank`, body `query` + `texts`) | + +VRAM ~2.7 GB for both, so ~13 GB is free. + +## Parity and speed vs esh-ml1 (2026-09-25 ~1540 PT) + +Client on nh3-dev, calling both seats directly. Corpus: 1,120 paragraphs from +this repo's docs plus 6 fixed texts (CJK, code, emoji, a 1-char input, a 6k-char +passage); 50 instruction-format queries; each host embedded everything twice. + +| embedding check | result | noise floor / control | +|---|---|---| +| per-text cosine, esh vs nh3 | median 0.999998, min 0.999993 | esh vs esh 0.999998 / 0.999995; nh3 vs nh3 0.999998 / 0.999993 | +| overlap@10, nh3 queries on the esh index | 1.000 | esh rerun 1.000; **positive control** MRL-256 truncation 0.684 | +| overlap@10, nh3 index + nh3 queries | 1.000 | — | +| hit@1 own paragraph | 0.76 (both hosts) | — | +| **negative control**, different texts | cosine median 0.50 | — | + +| rerank check (100 queries × 20 docs) | esh vs nh3 | esh vs esh | **positive control** (query cut to 4 words) | +|---|---|---|---| +| top-1 agreement | 1.00 | 1.00 | 0.94 | +| top-5 exact order | 0.98 | 0.97 | 0.09 | +| max score difference | 0.0014 | 0.0020 | 0.97 | + +**The two hosts cannot be told apart.** Every esh-vs-nh3 figure sits inside the +esh-vs-esh noise. Sensitivity floor: this method cannot resolve an embedding +cosine gap below ~5×10⁻⁶ or a rerank score difference below ~0.002. An index +built on either host serves queries from the other. + +**Speed, on-box**, 3 interleaved reps per host (range across reps): + +| workload | nh3-ml1 | esh-ml1 | +|---|---|---| +| embed 1 short query, p50 | 6.60–6.74 ms | 6.97–7.02 ms | +| bulk embed, 64 per request, passages/s | 103.7–105.0 | 105.5–108.1 | +| rerank 20 docs, p50 | 155.8–159.2 ms | 151.8–160.5 ms | + +The two hosts run at the same speed. esh-ml1 was also holding the idle reward seat +(~8 GB VRAM, 0% util) during these runs. + +**Gateway routing (Prime's call).** The recommendation is **load-share**. The +2026-09-25 rule against load-sharing came from pairing esh-ml1 with the much +faster fv-ml1. These two cards are identical, and a second site removes the +single-host outage the esh-ml1 README warns about. + +## How it is built + +1. [`playbooks/pve-nvidia-host.yaml`](../../playbooks/pve-nvidia-host.yaml) on + nh3-pve: driver **580.178.04** (open modules, DKMS) plus the + `nvidia-persistenced` unit. It needs **Secure Boot off**, which was turned off + in the BIOS on the 2026-09-25 visit; the pre-flight refuses otherwise. +2. [`playbooks/gpu-lxc.yaml`](../../playbooks/gpu-lxc.yaml) with the + "Run (nh3-ml1)" `--var` line from its header. + ⚠ nh3-pve was on **lxc-pve 6.0.0-1**. With it, every `docker run` in the CT + failed with *"open sysctl net.ipv4.ip_unprivileged_port_start file: reopen + fd 8: permission denied"* (runc 1.5 against the old AppArmor profile). + Upgrading lxc-pve alone to 6.0.0-2 (Proxmox fix #7006) and then running + `pct reboot 109` fixed it. The playbook now does the upgrade as its first step. +3. `scripts/deploy-stack.sh nh3-ml1 embed-rerank`, then set `HOST_NAME` / + `HOST_IP` in `.env`, then `docker compose up -d`. + +The **driver version lock** and the DKMS/kernel notes in +[esh-ml1's README](../esh-ml1/README.md#-driver-version-lock) apply here unchanged. +nh3-pve runs kernel 6.8.12-43, which the 2026-09-25 headers install pulled in +(see `servers/nh3-pve/README.md`). + +## Monitoring and telemetry (wired 2026-09-25) + +| layer | what | where | +|---|---|---| +| **Beszel** | NVIDIA agent `henrygd/beszel-agent-nvidia:0.18.7`, `stacks/beszel` + `hosts/nh3-ml1.yaml`, hub system `1feeeq61g4mkqre`; GPU util, VRAM and power are sampled | alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m | +| **Uptime Kuma** | `Embed — Qwen3 0.6B (TEI, nh3-ml1)` → `:8001/health` (#29); `Rerank — bge-v2-m3 (TEI, nh3-ml1)` → `:8013/health` (#30) | `stacks/uptimekuma/monitors.yaml` | +| **Homepage** | two cards under *AI - Eval & Retrieval*; dockerd on tcp/2375 bound to `10.100.50.80` | `stacks/homepage/conf/docker.yaml` → `nh3-ml1-docker` | +| **Dozzle** | agent `v10.4.1` on `10.100.50.80:7007`, compose dir `dozzle-agent`; added to the hub's `DOZZLE_REMOTE_AGENT` | hub on ana-docker :8088 | diff --git a/servers/nh3-ml1/ssh-target b/servers/nh3-ml1/ssh-target new file mode 100644 index 0000000..5e02652 --- /dev/null +++ b/servers/nh3-ml1/ssh-target @@ -0,0 +1 @@ +infra-ops@10.100.50.80 diff --git a/servers/nh3-ml1/system-details.txt b/servers/nh3-ml1/system-details.txt new file mode 100644 index 0000000..489b90d --- /dev/null +++ b/servers/nh3-ml1/system-details.txt @@ -0,0 +1,452 @@ + +===== HOST ===== + +Hostname: nh3-ml1 +Date: 2026-09-25T22:51:53+00:00 +Uptime: up 18 minutes +OS: Debian GNU/Linux 12 (bookworm) +Kernel: 6.8.12-43-pve +Arch: x86_64 + +===== HARDWARE ===== + +CPU cores: 6 +CPU model: 13th Gen Intel(R) Core(TM) i9-13900H +MemTotal: 16.0 GB +MemAvailable: 13.1 GB + +===== GPUS ===== + +index, name, memory.total [MiB], memory.free [MiB], driver_version +0, NVIDIA RTX 2000E Ada Generation, 16380 MiB, 13390 MiB, 580.178.04 + +===== FILESYSTEMS (df) ===== + +Filesystem Size Used Avail Use% Mounted on +rpool/data/subvol-109-disk-0 80G 12G 69G 15% / + +===== PERSISTENT MOUNTS (/etc/fstab, non-comment) ===== + + +===== TARGETED DATA PATHS ===== + +/opt (total: 3.2G) + total 11 + drwxr-xr-x 5 root root 5 2026-09-25 22:30 . + drwxr-xr-x 17 root root 21 2026-09-25 22:33 .. + drwxrwsr-x 4 root docker 4 2026-09-25 22:35 aimodels + drwx--x--x 4 root root 4 2026-09-25 22:30 containerd + drwxrwsr-x 4 root docker 4 2026-09-25 22:30 docker + +/opt/docker (total: 131K) + total 2 + drwxrwsr-x 4 root docker 4 2026-09-25 22:30 . + drwxr-xr-x 5 root root 5 2026-09-25 22:30 .. + drwxrwsr-x 5 root docker 5 2026-09-25 22:49 compose + drwxrwsr-x 2 root docker 2 2026-09-25 22:30 conf + +/opt/docker/compose (total: 130K) + total 27 + drwxrwsr-x 5 root docker 5 2026-09-25 22:49 . + drwxrwsr-x 4 root docker 4 2026-09-25 22:30 .. + drwxr-sr-x 4 infra-ops docker 8 2026-09-25 22:43 beszel + drwxr-sr-x 2 infra-ops docker 6 2026-09-25 22:49 dozzle-agent + drwxr-sr-x 2 infra-ops docker 6 2026-09-25 22:35 embed-rerank + +/opt/docker/conf (total: 512) + total 1 + drwxrwsr-x 2 root docker 2 2026-09-25 22:30 . + drwxrwsr-x 4 root docker 4 2026-09-25 22:30 .. + +/var/lib/docker (total: 8.5K) + +/srv (total: 512) + total 9 + drwxr-xr-x 2 root root 2 2025-09-07 15:14 . + drwxr-xr-x 17 root root 21 2026-09-25 22:33 .. + + +===== DOCKER ===== + +Server: 29.8.1 Client: 29.8.1 + +----- docker info ----- +Containers: 4 (running 4, paused 0, stopped 0) +Images: 4 +Runtimes: map[io.containerd.runc.v2:{{runc [] map[]} map[org.opencontainers.runtime-spec.features:{"ociVersionMin":"1.0.0","ociVersionMax":"1.3.0","hooks":["prestart","createRuntime","createContainer","startContainer","poststart","poststop"],"mountOptions":["async","atime","bind","defaults","dev","diratime","dirsync","exec","iversion","lazytime","loud","mand","noatime","nodev","nodiratime","noexec","noiversion","nolazytime","nomand","norelatime","nostrictatime","nosuid","nosymfollow","private","ratime","rbind","rdev","rdiratime","relatime","remount","rexec","rnoatime","rnodev","rnodiratime","rnoexec","rnorelatime","rnostrictatime","rnosuid","rnosymfollow","ro","rprivate","rrelatime","rro","rrw","rshared","rslave","rstrictatime","rsuid","rsymfollow","runbindable","rw","shared","silent","slave","strictatime","suid","symfollow","sync","tmpcopyup","unbindable"],"linux":{"namespaces":["cgroup","ipc","mount","network","pid","time","user","uts"],"capabilities":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_DAC_READ_SEARCH","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_SETGID","CAP_SETUID","CAP_SETPCAP","CAP_LINUX_IMMUTABLE","CAP_NET_BIND_SERVICE","CAP_NET_BROADCAST","CAP_NET_ADMIN","CAP_NET_RAW","CAP_IPC_LOCK","CAP_IPC_OWNER","CAP_SYS_MODULE","CAP_SYS_RAWIO","CAP_SYS_CHROOT","CAP_SYS_PTRACE","CAP_SYS_PACCT","CAP_SYS_ADMIN","CAP_SYS_BOOT","CAP_SYS_NICE","CAP_SYS_RESOURCE","CAP_SYS_TIME","CAP_SYS_TTY_CONFIG","CAP_MKNOD","CAP_LEASE","CAP_AUDIT_WRITE","CAP_AUDIT_CONTROL","CAP_SETFCAP","CAP_MAC_OVERRIDE","CAP_MAC_ADMIN","CAP_SYSLOG","CAP_WAKE_ALARM","CAP_BLOCK_SUSPEND","CAP_AUDIT_READ","CAP_PERFMON","CAP_BPF","CAP_CHECKPOINT_RESTORE"],"cgroup":{"v1":true,"v2":true,"systemd":true,"systemdUser":true,"rdma":true},"seccomp":{"enabled":true,"actions":["SCMP_ACT_ALLOW","SCMP_ACT_ERRNO","SCMP_ACT_KILL","SCMP_ACT_KILL_PROCESS","SCMP_ACT_KILL_THREAD","SCMP_ACT_LOG","SCMP_ACT_NOTIFY","SCMP_ACT_TRACE","SCMP_ACT_TRAP"],"operators":["SCMP_CMP_EQ","SCMP_CMP_GE","SCMP_CMP_GT","SCMP_CMP_LE","SCMP_CMP_LT","SCMP_CMP_MASKED_EQ","SCMP_CMP_NE"],"archs":["SCMP_ARCH_AARCH64","SCMP_ARCH_ARM","SCMP_ARCH_LOONGARCH64","SCMP_ARCH_MIPS","SCMP_ARCH_MIPS64","SCMP_ARCH_MIPS64N32","SCMP_ARCH_MIPSEL","SCMP_ARCH_MIPSEL64","SCMP_ARCH_MIPSEL64N32","SCMP_ARCH_PPC","SCMP_ARCH_PPC64","SCMP_ARCH_PPC64LE","SCMP_ARCH_RISCV64","SCMP_ARCH_S390","SCMP_ARCH_S390X","SCMP_ARCH_X32","SCMP_ARCH_X86","SCMP_ARCH_X86_64"],"knownFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG","SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV"],"supportedFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"]},"apparmor":{"enabled":true},"selinux":{"enabled":true},"intelRdt":{"enabled":true,"schemata":true,"monitoring":true},"memoryPolicy":{"modes":["MPOL_BIND","MPOL_DEFAULT","MPOL_INTERLEAVE","MPOL_LOCAL","MPOL_PREFERRED","MPOL_PREFERRED_MANY","MPOL_WEIGHTED_INTERLEAVE"],"flags":["MPOL_F_NUMA_BALANCING","MPOL_F_RELATIVE_NODES","MPOL_F_STATIC_NODES"]},"mountExtensions":{"idmap":{"enabled":true}},"netDevices":{"enabled":true}},"annotations":{"io.github.seccomp.libseccomp.version":"2.5.4","org.opencontainers.runc.checkpoint.enabled":"true","org.opencontainers.runc.commit":"v1.5.1-0-g8f2685a4","org.opencontainers.runc.version":"1.5.1"},"potentiallyUnsafeConfigAnnotations":["bundle","org.systemd.property.","org.criu.config"]}]} nvidia:{{nvidia-container-runtime [] map[]} map[org.opencontainers.runtime-spec.features:{"ociVersionMin":"1.0.0","ociVersionMax":"1.3.0","hooks":["prestart","createRuntime","createContainer","startContainer","poststart","poststop"],"mountOptions":["async","atime","bind","defaults","dev","diratime","dirsync","exec","iversion","lazytime","loud","mand","noatime","nodev","nodiratime","noexec","noiversion","nolazytime","nomand","norelatime","nostrictatime","nosuid","nosymfollow","private","ratime","rbind","rdev","rdiratime","relatime","remount","rexec","rnoatime","rnodev","rnodiratime","rnoexec","rnorelatime","rnostrictatime","rnosuid","rnosymfollow","ro","rprivate","rrelatime","rro","rrw","rshared","rslave","rstrictatime","rsuid","rsymfollow","runbindable","rw","shared","silent","slave","strictatime","suid","symfollow","sync","tmpcopyup","unbindable"],"linux":{"namespaces":["cgroup","ipc","mount","network","pid","time","user","uts"],"capabilities":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_DAC_READ_SEARCH","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_SETGID","CAP_SETUID","CAP_SETPCAP","CAP_LINUX_IMMUTABLE","CAP_NET_BIND_SERVICE","CAP_NET_BROADCAST","CAP_NET_ADMIN","CAP_NET_RAW","CAP_IPC_LOCK","CAP_IPC_OWNER","CAP_SYS_MODULE","CAP_SYS_RAWIO","CAP_SYS_CHROOT","CAP_SYS_PTRACE","CAP_SYS_PACCT","CAP_SYS_ADMIN","CAP_SYS_BOOT","CAP_SYS_NICE","CAP_SYS_RESOURCE","CAP_SYS_TIME","CAP_SYS_TTY_CONFIG","CAP_MKNOD","CAP_LEASE","CAP_AUDIT_WRITE","CAP_AUDIT_CONTROL","CAP_SETFCAP","CAP_MAC_OVERRIDE","CAP_MAC_ADMIN","CAP_SYSLOG","CAP_WAKE_ALARM","CAP_BLOCK_SUSPEND","CAP_AUDIT_READ","CAP_PERFMON","CAP_BPF","CAP_CHECKPOINT_RESTORE"],"cgroup":{"v1":true,"v2":true,"systemd":true,"systemdUser":true,"rdma":true},"seccomp":{"enabled":true,"actions":["SCMP_ACT_ALLOW","SCMP_ACT_ERRNO","SCMP_ACT_KILL","SCMP_ACT_KILL_PROCESS","SCMP_ACT_KILL_THREAD","SCMP_ACT_LOG","SCMP_ACT_NOTIFY","SCMP_ACT_TRACE","SCMP_ACT_TRAP"],"operators":["SCMP_CMP_EQ","SCMP_CMP_GE","SCMP_CMP_GT","SCMP_CMP_LE","SCMP_CMP_LT","SCMP_CMP_MASKED_EQ","SCMP_CMP_NE"],"archs":["SCMP_ARCH_AARCH64","SCMP_ARCH_ARM","SCMP_ARCH_LOONGARCH64","SCMP_ARCH_MIPS","SCMP_ARCH_MIPS64","SCMP_ARCH_MIPS64N32","SCMP_ARCH_MIPSEL","SCMP_ARCH_MIPSEL64","SCMP_ARCH_MIPSEL64N32","SCMP_ARCH_PPC","SCMP_ARCH_PPC64","SCMP_ARCH_PPC64LE","SCMP_ARCH_RISCV64","SCMP_ARCH_S390","SCMP_ARCH_S390X","SCMP_ARCH_X32","SCMP_ARCH_X86","SCMP_ARCH_X86_64"],"knownFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG","SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV"],"supportedFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"]},"apparmor":{"enabled":true},"selinux":{"enabled":true},"intelRdt":{"enabled":true,"schemata":true,"monitoring":true},"memoryPolicy":{"modes":["MPOL_BIND","MPOL_DEFAULT","MPOL_INTERLEAVE","MPOL_LOCAL","MPOL_PREFERRED","MPOL_PREFERRED_MANY","MPOL_WEIGHTED_INTERLEAVE"],"flags":["MPOL_F_NUMA_BALANCING","MPOL_F_RELATIVE_NODES","MPOL_F_STATIC_NODES"]},"mountExtensions":{"idmap":{"enabled":true}},"netDevices":{"enabled":true}},"annotations":{"io.github.seccomp.libseccomp.version":"2.5.4","org.opencontainers.runc.checkpoint.enabled":"true","org.opencontainers.runc.commit":"v1.5.1-0-g8f2685a4","org.opencontainers.runc.version":"1.5.1"},"potentiallyUnsafeConfigAnnotations":["bundle","org.systemd.property.","org.criu.config"]}]} runc:{{runc [] map[]} map[org.opencontainers.runtime-spec.features:{"ociVersionMin":"1.0.0","ociVersionMax":"1.3.0","hooks":["prestart","createRuntime","createContainer","startContainer","poststart","poststop"],"mountOptions":["async","atime","bind","defaults","dev","diratime","dirsync","exec","iversion","lazytime","loud","mand","noatime","nodev","nodiratime","noexec","noiversion","nolazytime","nomand","norelatime","nostrictatime","nosuid","nosymfollow","private","ratime","rbind","rdev","rdiratime","relatime","remount","rexec","rnoatime","rnodev","rnodiratime","rnoexec","rnorelatime","rnostrictatime","rnosuid","rnosymfollow","ro","rprivate","rrelatime","rro","rrw","rshared","rslave","rstrictatime","rsuid","rsymfollow","runbindable","rw","shared","silent","slave","strictatime","suid","symfollow","sync","tmpcopyup","unbindable"],"linux":{"namespaces":["cgroup","ipc","mount","network","pid","time","user","uts"],"capabilities":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_DAC_READ_SEARCH","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_SETGID","CAP_SETUID","CAP_SETPCAP","CAP_LINUX_IMMUTABLE","CAP_NET_BIND_SERVICE","CAP_NET_BROADCAST","CAP_NET_ADMIN","CAP_NET_RAW","CAP_IPC_LOCK","CAP_IPC_OWNER","CAP_SYS_MODULE","CAP_SYS_RAWIO","CAP_SYS_CHROOT","CAP_SYS_PTRACE","CAP_SYS_PACCT","CAP_SYS_ADMIN","CAP_SYS_BOOT","CAP_SYS_NICE","CAP_SYS_RESOURCE","CAP_SYS_TIME","CAP_SYS_TTY_CONFIG","CAP_MKNOD","CAP_LEASE","CAP_AUDIT_WRITE","CAP_AUDIT_CONTROL","CAP_SETFCAP","CAP_MAC_OVERRIDE","CAP_MAC_ADMIN","CAP_SYSLOG","CAP_WAKE_ALARM","CAP_BLOCK_SUSPEND","CAP_AUDIT_READ","CAP_PERFMON","CAP_BPF","CAP_CHECKPOINT_RESTORE"],"cgroup":{"v1":true,"v2":true,"systemd":true,"systemdUser":true,"rdma":true},"seccomp":{"enabled":true,"actions":["SCMP_ACT_ALLOW","SCMP_ACT_ERRNO","SCMP_ACT_KILL","SCMP_ACT_KILL_PROCESS","SCMP_ACT_KILL_THREAD","SCMP_ACT_LOG","SCMP_ACT_NOTIFY","SCMP_ACT_TRACE","SCMP_ACT_TRAP"],"operators":["SCMP_CMP_EQ","SCMP_CMP_GE","SCMP_CMP_GT","SCMP_CMP_LE","SCMP_CMP_LT","SCMP_CMP_MASKED_EQ","SCMP_CMP_NE"],"archs":["SCMP_ARCH_AARCH64","SCMP_ARCH_ARM","SCMP_ARCH_LOONGARCH64","SCMP_ARCH_MIPS","SCMP_ARCH_MIPS64","SCMP_ARCH_MIPS64N32","SCMP_ARCH_MIPSEL","SCMP_ARCH_MIPSEL64","SCMP_ARCH_MIPSEL64N32","SCMP_ARCH_PPC","SCMP_ARCH_PPC64","SCMP_ARCH_PPC64LE","SCMP_ARCH_RISCV64","SCMP_ARCH_S390","SCMP_ARCH_S390X","SCMP_ARCH_X32","SCMP_ARCH_X86","SCMP_ARCH_X86_64"],"knownFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG","SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV"],"supportedFlags":["SECCOMP_FILTER_FLAG_TSYNC","SECCOMP_FILTER_FLAG_SPEC_ALLOW","SECCOMP_FILTER_FLAG_LOG"]},"apparmor":{"enabled":true},"selinux":{"enabled":true},"intelRdt":{"enabled":true,"schemata":true,"monitoring":true},"memoryPolicy":{"modes":["MPOL_BIND","MPOL_DEFAULT","MPOL_INTERLEAVE","MPOL_LOCAL","MPOL_PREFERRED","MPOL_PREFERRED_MANY","MPOL_WEIGHTED_INTERLEAVE"],"flags":["MPOL_F_NUMA_BALANCING","MPOL_F_RELATIVE_NODES","MPOL_F_STATIC_NODES"]},"mountExtensions":{"idmap":{"enabled":true}},"netDevices":{"enabled":true}},"annotations":{"io.github.seccomp.libseccomp.version":"2.5.4","org.opencontainers.runc.checkpoint.enabled":"true","org.opencontainers.runc.commit":"v1.5.1-0-g8f2685a4","org.opencontainers.runc.version":"1.5.1"},"potentiallyUnsafeConfigAnnotations":["bundle","org.systemd.property.","org.criu.config"]}]}] +Default runtime: runc +Storage driver: overlayfs +Root dir: /var/lib/docker +Server version: 29.8.1 + +----- running containers ----- +NAMES IMAGE STATUS PORTS +dozzle-agent amir20/dozzle:v10.4.1 Up About a minute 10.100.50.80:7007->7007/tcp, 8080/tcp +beszel-agent henrygd/beszel-agent-nvidia:0.18.7 Up 7 minutes (healthy) +tei-rerank ghcr.io/huggingface/text-embeddings-inference:89-1.9.4 Up 15 minutes (healthy) 0.0.0.0:8013->80/tcp, [::]:8013->80/tcp +tei-embed ghcr.io/huggingface/text-embeddings-inference:89-1.9.4 Up 15 minutes (healthy) 0.0.0.0:8001->80/tcp, [::]:8001->80/tcp + +----- all containers ----- +NAMES IMAGE STATUS +dozzle-agent amir20/dozzle:v10.4.1 Up About a minute +beszel-agent henrygd/beszel-agent-nvidia:0.18.7 Up 7 minutes (healthy) +tei-rerank ghcr.io/huggingface/text-embeddings-inference:89-1.9.4 Up 15 minutes (healthy) +tei-embed ghcr.io/huggingface/text-embeddings-inference:89-1.9.4 Up 15 minutes (healthy) + +----- networks ----- +NAME DRIVER SCOPE +bridge bridge local +embed-rerank_default bridge local +host host local +none null local +traefik-net bridge local + +----- networks (external, non-default — worth knowing for compose external: true) ----- +embed-rerank_default +traefik-net + +----- named volumes ----- +VOLUME NAME DRIVER +beszel_beszel_agent_data local +dozzle-agent_dozzle_agent_data local + +----- compose projects currently running ----- +beszel +dozzle-agent +embed-rerank + +===== COMPOSE FILES (/opt/docker/compose/) ===== + + +>>> /opt/docker/compose/beszel/compose.yaml +# Beszel — lightweight server/container monitoring. +# +# Hub: single web UI with the SQLite store. Agents: per-host metric collectors +# that the hub pulls from over SSH. +# +# Multi-host layout via compose profiles: +# COMPOSE_PROFILES=hub → hub only (ana-docker) +# COMPOSE_PROFILES=hub,agent → hub + local agent on the same host +# COMPOSE_PROFILES=agent → agent only (ana-ml2, nh3-docker, +# esh-docker-vm, vm-esh-nas) +# +# The agent uses network_mode: host so it sees real host CPU/mem/net/disk +# counters rather than container-scoped ones — that's why it can't share +# the tnet network with the hub. +# +# All tunables live in .env — edit that, not this file. + +services: + beszel: + image: henrygd/beszel:${BESZEL_VERSION} + container_name: beszel + profiles: [hub] + restart: unless-stopped + ports: + - "${BESZEL_PORT}:8090" + volumes: + - beszel_data:/beszel_data + healthcheck: + # Hub image is distroless — no wget/curl. Use the bundled `/beszel` + # binary's built-in health subcommand (https://beszel.dev/guide/healthchecks). + test: ["CMD", "/beszel", "health", "--url", "http://localhost:8090"] + interval: 120s + timeout: 10s + retries: 3 + start_period: 15s + networks: + - tnet + labels: + - homepage.group=Monitoring + - homepage.name=Beszel + - homepage.icon=mdi-chart-line + - homepage.description=Server + container monitoring + - homepage.href=http://10.250.50.70:${BESZEL_PORT} + - homepage.widget.type=beszel + - homepage.widget.url=http://10.250.50.70:${BESZEL_PORT} + - homepage.widget.version=2 + - homepage.widget.username={{HOMEPAGE_VAR_BESZEL_USERNAME}} + - homepage.widget.password={{HOMEPAGE_VAR_BESZEL_PASSWORD}} + + beszel-agent: + image: henrygd/beszel-agent:${BESZEL_VERSION} + container_name: beszel-agent + profiles: [agent] + restart: unless-stopped + network_mode: host + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - beszel_agent_data:/var/lib/beszel-agent + environment: + # Agent auth has two modes (v0.13+ supports both side-by-side): + # - KEY-mode: agent listens, hub connects inbound over SSH using KEY. + # Requires BESZEL_HUB_KEY in .env. + # - Token-mode: agent initiates an outbound connection to HUB_URL + # using TOKEN. Easier through NAT. Requires HUB_URL + BESZEL_TOKEN. + # Leave unused ones empty ("") in .env; both can be set simultaneously. + - PORT=${BESZEL_AGENT_PORT:-45876} + - KEY=${BESZEL_HUB_KEY:-} + - HUB_URL=${HUB_URL:-} + - TOKEN=${BESZEL_TOKEN:-} + - EXTRA_FILESYSTEMS=${BESZEL_EXTRA_FS:-} + healthcheck: + # Agent image ships the `/agent` binary with a `health` subcommand. + # Verifies the agent process is up — not that the hub can reach it. + test: ["CMD", "/agent", "health"] + interval: 120s + timeout: 10s + retries: 3 + start_period: 15s + +volumes: + beszel_data: + beszel_agent_data: + +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/beszel/synology/compose.yaml +services: + beszel-agent: + image: henrygd/beszel-agent:0.18.7 + container_name: beszel-agent + restart: unless-stopped + network_mode: host + volumes: + - ./agent-data:/var/lib/beszel-agent + - /var/run/docker.sock:/var/run/docker.sock:ro + - /usr/share/zoneinfo:/extra-filesystems/system:ro + - ./:/extra-filesystems/volume1:ro + environment: + PORT: '45876' + KEY: 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt' + FILESYSTEM: /extra-filesystems/system + EXTRA_FILESYSTEMS: /extra-filesystems/volume1 + +>>> /opt/docker/compose/dozzle-agent/compose.yaml +# Dozzle — container log viewer. +# +# Multi-host layout via compose profiles: +# COMPOSE_PROFILES=hub → runs the web UI (deploy on ana-docker) +# COMPOSE_PROFILES=agent → runs the remote agent (deploy on ana-ml2) +# +# Same compose.yaml on both servers; per-host `.env` picks the profile. +# +# All tunables live in .env — edit that, not this file. + +services: + dozzle: + image: amir20/dozzle:${DOZZLE_VERSION} + container_name: dozzle + profiles: [hub] + restart: unless-stopped + ports: + - "${DOZZLE_PORT}:8080" + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - dozzle_data:/data + environment: + - DOZZLE_HOSTNAME=${DOZZLE_HOSTNAME} + - DOZZLE_REMOTE_AGENT=${DOZZLE_REMOTE_AGENT:-} + - DOZZLE_AUTH_PROVIDER=${DOZZLE_AUTH_PROVIDER:-none} + - DOZZLE_USERNAME=${DOZZLE_USERNAME:-} + - DOZZLE_PASSWORD=${DOZZLE_PASSWORD:-} + healthcheck: + test: ["CMD", "/dozzle", "healthcheck"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 15s + networks: + - tnet + labels: + - homepage.group=Monitoring + - homepage.name=Dozzle + - homepage.icon=mdi-text-box-search + - homepage.description=Container logs (ana-docker + ana-ml2) + - homepage.href=http://10.250.50.70:${DOZZLE_PORT} + + dozzle-agent: + image: amir20/dozzle:${DOZZLE_VERSION} + container_name: dozzle-agent + profiles: [agent] + restart: unless-stopped + command: agent + ports: + - "${DOZZLE_AGENT_BIND:-0.0.0.0}:${DOZZLE_AGENT_PORT}:7007" + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - dozzle_agent_data:/data + environment: + - DOZZLE_HOSTNAME=${DOZZLE_HOSTNAME} + networks: + - tnet + +volumes: + dozzle_data: + dozzle_agent_data: + +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/embed-rerank/compose.yaml +# embed-rerank — THE fleet's embedding + reranking service, on esh-ml1 (CT 110 on +# esh-pve, RTX 2000E Ada, 16 GB). Served by Hugging Face Text Embeddings +# Inference (TEI). A second instance runs on nh3-ml1 (CT 109 on nh3-pve, the same +# card) since 2026-09-25; the per-host bits are HOST_NAME / HOST_IP in .env, and +# their defaults are esh-ml1's, so esh-ml1's live .env needs no change. +# +# Prime, 2026-09-25: "TEI is embed/reranker server for esh-ml1 and the FLEET in +# general, in future." It replaced vLLM here the same day, after a side-by-side +# bake-off on this card (docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md). TEI +# gives the same vectors as the old vLLM seats (no re-embedding), but it is +# ~1.3x slower on bulk work on this card. It is much lighter (2.6 GB VRAM for +# both, 8 GB image, ~4 s restart). fv-ml1's vLLM embed/rerank seats were +# retired after this went live. +# +# tei-embed Qwen/Qwen3-Embedding-0.6B → /v1/embeddings (OpenAI), /embed :8001 +# tei-rerank BAAI/bge-reranker-v2-m3 → /rerank (body: query + texts) :8013 +# +# Ports kept from the vLLM era (and fv-ml1), so the embedding gateway entry did +# not change address. ⚠ The RERANK gateway entry must use LiteLLM's +# `huggingface/` provider: `hosted_vllm/` sends `documents` and TEI answers 422 +# "missing field texts". +# +# ⚠ Embedding vectors are model-specific. Never change EMBED_MODEL without a +# re-embedding plan for every index built on it (Worldtree, nevermore, Open WebUI). +# +# FAIL-CLOSED truncation (--auto-truncate false). TEI's default silently +# embedded the first 16,384 tokens of a ~40k-token input and returned 200. +# Turning it off requires --max-batch-tokens >= the model's max input (32,768 +# for Qwen3-Embedding), or TEI refuses to start. +# +# NO `tnet`/traefik-net: esh-ml1 runs no traefik; consumers reach the published +# ports, and in practice only the LiteLLM gateway does (verified from the seats' +# logs 2026-09-25: every request matched a gateway spend-log row). +# +# Host setup: playbooks/pve-nvidia-host.yaml, then playbooks/gpu-lxc.yaml. +# Tunables live in .env. + +name: embed-rerank + +services: + tei-embed: + image: ghcr.io/huggingface/text-embeddings-inference:${TEI_TAG} + container_name: tei-embed + restart: unless-stopped + ports: + - "${EMBED_PORT}:80" + volumes: + - /opt/aimodels/tei-cache:/data + environment: + - HF_TOKEN=${HF_TOKEN:-} + command: + - --model-id + - ${EMBED_MODEL} + - --served-model-name + - ${EMBED_MODEL} + # TEI on CUDA is float16-only; parity vs the bf16 vLLM vectors was measured. + - --dtype + - float16 + # Default 32; vLLM had no cap and callers batch 64. + - --max-client-batch-size + - "${MAX_CLIENT_BATCH_SIZE}" + - --auto-truncate + - "false" + - --max-batch-tokens + - "32768" + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: ["0"] + capabilities: [gpu] + healthcheck: + test: ["CMD", "curl", "-fsS", "http://localhost:80/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s + labels: + - homepage.group=AI - Eval & Retrieval + - homepage.name=Embed — Qwen3 0.6B (TEI, ${HOST_NAME:-esh-ml1}) + - homepage.icon=mdi-vector-arrange-below + - homepage.description=Fleet embeddings (qwen3-embedding) via TEI on ${HOST_NAME:-esh-ml1} + - homepage.href=http://${HOST_IP:-10.0.50.80}:${EMBED_PORT}/docs + + tei-rerank: + image: ghcr.io/huggingface/text-embeddings-inference:${TEI_TAG} + container_name: tei-rerank + restart: unless-stopped + ports: + - "${RERANK_PORT}:80" + volumes: + - /opt/aimodels/tei-cache:/data + environment: + - HF_TOKEN=${HF_TOKEN:-} + command: + - --model-id + - ${RERANK_MODEL} + - --dtype + - float16 + - --max-client-batch-size + - "${MAX_CLIENT_BATCH_SIZE}" + # Fail-closed; bge-reranker-v2-m3's max input (8,192) fits the default + # max-batch-tokens (16,384). + - --auto-truncate + - "false" + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: ["0"] + capabilities: [gpu] + healthcheck: + test: ["CMD", "curl", "-fsS", "http://localhost:80/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s + labels: + - homepage.group=AI - Eval & Retrieval + - homepage.name=Rerank — bge-v2-m3 (TEI, ${HOST_NAME:-esh-ml1}) + - homepage.icon=mdi-sort-variant + - homepage.description=Fleet reranker (reranker) via TEI on ${HOST_NAME:-esh-ml1} + - homepage.href=http://${HOST_IP:-10.0.50.80}:${RERANK_PORT}/docs + +===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) ===== + +/opt/docker/conf + +===== LISTENING PORTS ===== + +*:22 +*:45876 +0.0.0.0:8001 +0.0.0.0:8013 +10.100.50.80:2375 +10.100.50.80:7007 +127.0.0.1:25 +[::1]:25 +[::]:8001 +[::]:8013 + +===== MODEL / HUGGINGFACE CACHES ===== + + +===== DOCKER-ADJACENT SYSTEMD SERVICES ===== + +container-getty@1.service running +container-getty@2.service running +containerd.service running +docker.service running + +===== DONE ===== + +Paste the above back into the chat, or pass a path as argv[1] to save. diff --git a/servers/nh3-pve/README.md b/servers/nh3-pve/README.md index 9e032c9..0190d6c 100644 --- a/servers/nh3-pve/README.md +++ b/servers/nh3-pve/README.md @@ -13,7 +13,7 @@ Proxmox VE hypervisor for the NH3 site (`nh3-vmhost.phasefinal.com`). - **CPU:** 13th Gen Intel Core i9-13900H - **RAM:** 62.5 GB -- **Kernel:** `6.8.12-11-pve` (Proxmox 8.x) +- **Kernel:** `6.8.12-43-pve` since 2026-09-25 (was `6.8.12-11`; see the kernel bullet below). PVE `8.4.1`, well behind esh-pve's `8.4.20` (177 packages pending) - **Storage:** mostly networked — `/mnt/pve/pfi-nh3-nas` (42 TB) mounted from the Synology at `10.100.50.50:/volume1/VMStorage`; ~27 TB used ## What it runs @@ -33,6 +33,7 @@ after a power loss** is the column that matters in a recovery: a guest listed as | 103 | nh3-wg | CT | 1 | up | | 106 | nh3-headscale | CT | 1 | up | | 107 | nh3-scale | CT | 1 | up (mesh subnet router + fleet egress proxy) | +| 109 | nh3-ml1 (`10.100.50.80`) | CT | 1 (order 30) | up. GPU LXC, second embed/rerank backend (`servers/nh3-ml1/README.md`); needs the NVIDIA module, so it is the one guest a driver fault can stop | **Power-loss recovery (2026-09-24 outage).** Every guest boots at once, and nh3-nas is the slowest to serve NFS. NFS clients now mount nh3-nas shares on @@ -55,12 +56,19 @@ not power back on by itself. vmbr0 members, so either cage works. **Never drop either port from the bridge** without checking which one has carrier (`ip -br link`). The bridge carries the I226-V's MAC `…:96:0d` because it is the first port listed. -- **AMT: NOT wired.** The AMT-capable I226-LM (`enp88s0`, MAC `58:47:ca:76:96:0e`) - has no cable. The ME is present (`/dev/mei0`, "AMT SOL Redirection" 00:16.3), but - MEBx provisioning status is unknown. To wire it: cable the LM port, then at boot - press Ctrl+P → set the MEBx password, enable manageability, set network (static - or DHCP), KVM on, User Opt-in = None, activate network access. MEBx can be driven - remotely through the NanoKVM below. +- **AMT: cabled and enabled, but NOT reachable (2026-09-25 visit).** The I226-LM + (`enp88s0`, MAC `58:47:ca:76:96:0e`) now has a 1 Gb link (measured at 1552 by + bringing the port up unbridged for a few seconds). Prime reports AMT enabled in + MEBx. **Nothing answers on the network, though.** The UDM has no client or lease + for `…:96:0e`, and no host on `10.100.{0,10,50,250}.0/24` has 16992 or 16993 + open. The sweep was checked against known-open ports and does see them. The + cable is not on the USW Pro 24, whose up ports are 19, 22, 23 and 26, all + accounted for. So it is on another switch, most likely nh3-sw1 + (`10.100.250.2`, no infra-ops access). Likely causes: MEBx "Activate Network + Access" was not done, a static IP outside those subnets, or a switch port on a + VLAN that gets no DHCP. MEBx menu: Ctrl+P at boot → password, manageability on, + network (static or DHCP), KVM on, User Opt-in = None, activate network access. + **Keep the NanoKVM here until AMT KVM is confirmed.** - **Console OOB exists: a Sipeed NanoKVM** is attached (USB `3346:1009` on the host; web UI **`https://10.100.250.171`**, switch port 23, nh3-mgmt). It gives video and keyboard, so BIOS, MEBx and a host that booted without network are all reachable @@ -80,48 +88,42 @@ not power back on by itself. - **`enp88s0` (the AMT port) is no longer a vmbr0 bridge port** (file edited 2026-09-25, effective next boot). STP is off, so bridging a second cabled uplink into the same L2 would loop the site LAN. -- **GPU installed 2026-09-25: RTX 2000E Ada at `01:00.0`** (`10de:28b0`). The pins - held: the X710 moved to bus 03 and every NIC kept its name. No NVIDIA driver yet, - so `nouveau` binds it (`gsp ctor failed: -2` is expected without GSP firmware). - ⚠ **After the install the iGPU is gone from the PCI bus**: `00:02.0` enumerated on - the 2026-09-24 boot and is absent now, `/dev/dri` does not exist, and the RTX is - `boot_vga=1`. The BIOS's Auto primary display picked the PCIe card and hid the - iGPU, so the NanoKVM's iGPU-HDMI capture has no source. Fix: BIOS → Primary - Display = IGFX (or enable iGPU Multi-Monitor). Reaching the BIOS now needs a - display on the RTX's mini-DP, for example the NanoKVM through a mini-DP→HDMI - adapter. Verify with `lspci | grep 00:02.0` and `boot_vga` on `00:02.0`. -- ⚠ **Secure Boot is ON here** (`mokutil --sb-state`, lockdown `integrity`; - enrolled MOK = the Proxmox Secure Boot CA). esh-pve, the same model, has it - OFF. Any DKMS-built module, the NVIDIA driver included, is refused until its - key is enrolled through MokManager at boot or Secure Boot is turned off in the - BIOS. Both need the console, and the console is blind (below). The 2026-09-25 - NVIDIA install failed on this and rolled itself back. What it left, all - harmless: headers `6.8.12-11` plus the series meta, dkms and build-essential; - nouveau blacklisted and unloaded; the `.run` staged in `/root/nvidia`. -- **OOB decision (Prime via Miranda, 2026-09-25): HOLD. nh3-pve stays console-blind - until the next NH3 site visit.** No BIOS change and no reboot until then. The - accepted risk: a boot without network means a site trip. **Target end state:** - the NanoKVM moves to pfi-gx10, and this MS-01 gets out-of-band access through its - own vPro/AMT (cable the I226-LM `enp88s0` and provision MEBx; see the AMT bullet - above). - ⚠ **Do the BIOS fix on that same visit.** AMT's KVM redirection captures only - the Intel iGPU's framebuffer, so with the iGPU hidden as it is now, AMT KVM would - be just as blind as the NanoKVM. Order: set Primary Display = IGFX, provision - MEBx, confirm AMT KVM shows the console, and only then move the NanoKVM to the - gx10. - **Reaching the BIOS** (2026-09-25 facts). The NanoKVM input is HDMI, the RTX has - mini-DP only, and the adapters on hand are mini-DP→DP only. The paths: - (1) an **active** mini-DP→HDMI adapter to feed the NanoKVM, since a passive one - depends on DP++; (2) a DisplayPort monitor plus a USB keyboard at the rack; - (3) pull the card, boot on the iGPU, set IGFX explicitly (not Auto), then refit - the card. The NIC pins make a card-out boot safe. - `systemctl reboot --firmware-setup` works here (`OsIndicationsSupported` bit 0 - set), so nobody has to catch the Del key at POST. - **An OS-side patch is not possible:** the AMI `Setup`/`SaSetup` variables are not - runtime-visible here, unlike on the gx10. - **Known-good reference: esh-pve.** Same MS-01, same BIOS `AHWSA.1.17`, same RTX - 2000E, and its `00:02.0` is present with `boot_vga=1`. The target state works on - this hardware. +- **GPU: RTX 2000E Ada at `01:00.0`** (`10de:28b0`), installed 2026-09-25. The + NIC pins held: the X710 moved to bus 03 and every NIC kept its name. **NVIDIA + 580.178.04** (open modules, DKMS) has been on the host since 2026-09-25 at 1527 + (`playbooks/pve-nvidia-host.yaml`), with `nvidia-persistenced` ordered before + `pve-guests`. It serves CT 109 nh3-ml1. +- **iGPU restored (2026-09-25 visit).** With the card in, the BIOS's Auto primary + display had hidden the iGPU. That left the NanoKVM (iGPU HDMI) blind and would + have blinded AMT KVM too, since AMT captures only the iGPU. It was set on the + visit, and since the 1523 boot `00:02.0` is back with `boot_vga=1` and i915 + loaded. The NanoKVM should have video again (not checked from here). + `systemctl reboot --firmware-setup` works (`OsIndicationsSupported` bit 0), so + nobody has to catch Del at POST. The AMI `Setup` variables are not + runtime-visible, so there is no OS-side BIOS patch. esh-pve, the same MS-01 and + BIOS `AHWSA.1.17`, is the known-good reference. +- **Secure Boot: OFF since the 2026-09-25 visit** (`mokutil --sb-state`: disabled), + which matches esh-pve. While it was ON (lockdown `integrity`), the DKMS NVIDIA + module was refused and the first install rolled itself back. The playbook's + pre-flight refuses if it is ever turned back on without an enrolled DKMS MOK. +- ⚠ **Kernel jumped `6.8.12-11` → `6.8.12-43` at the visit reboot, pulled in by + our own playbook.** At 1419 the headers step ran `apt-get install + proxmox-headers-6.8`. That upgraded the `proxmox-kernel-6.8` meta and installed + `proxmox-kernel-6.8.12-43-pve-signed`. Nobody chose the new kernel; it booted + because it was the newest. + Side effect: **every -43 boot oopses in Bluetooth** (`btmtk_usb_hci_wmt_sync` → + NULL deref in `hci_power_on`, the MS-01's MediaTek BT; taint `D`). It hit on all + three -43 boots and on none of the -11 boots. esh-pve on `-42` shows the same + oops and has run fine, so it is benign so far: only the BT worker dies. The fix + is to blacklist `btusb` on both hypervisors. That is not done, because it only + takes effect at the next boot. +- **lxc-pve 6.0.0-1 → 6.0.0-2** (2026-09-25 1533, that one package only). This is + Proxmox fix #7006. Without it, runc 1.5 inside a nesting CT fails every + `docker run`. `playbooks/gpu-lxc.yaml` now upgrades it first. +- **OOB plan status.** Prime ruled on 2026-09-25 via Miranda to HOLD console-blind + until the site visit. Target: the NanoKVM moves to pfi-gx10, and this MS-01 uses + its own AMT. The visit did IGFX, turned SB off and cabled plus enabled AMT. + **Still open:** AMT is not on the network (above), so the NanoKVM stays here. ## Refresh state diff --git a/stacks/beszel/README.md b/stacks/beszel/README.md index c04af76..11a65dd 100644 --- a/stacks/beszel/README.md +++ b/stacks/beszel/README.md @@ -24,6 +24,7 @@ filesystem samples verified; fleet 13/14 up with known fv-ml1 outage. | vm-esh-nas | beszel-agent-esh-nas | /mnt/books, /mnt/share, /mnt/music, /mnt/media | | nh3-dev | beszel | none since 2026-09-25 (was /mnt/backup, /mnt/smithy — see below) | | esh-ml1 (added 2026-09-25) | beszel | none — NVIDIA image (`hosts/esh-ml1.yaml`) for the RTX 2000E Ada | +| nh3-ml1 (added 2026-09-25) | beszel | none — NVIDIA image (`hosts/nh3-ml1.yaml`) for the RTX 2000E Ada; hub system `1feeeq61g4mkqre`, same 5 alerts as esh-ml1 | ⚠ **nh3-dev's agent was DOWN from the 2026-09-24 NH3 power recovery until 2026-09-25.** Docker could not bind `/mnt/smithy` at boot ("no such device"): since diff --git a/stacks/beszel/hosts/nh3-ml1.yaml b/stacks/beszel/hosts/nh3-ml1.yaml new file mode 100644 index 0000000..6c27a50 --- /dev/null +++ b/stacks/beszel/hosts/nh3-ml1.yaml @@ -0,0 +1,15 @@ +# nh3-ml1 (CT 109 on nh3-pve) — Beszel agent with NVIDIA GPU telemetry for the +# RTX 2000E Ada (utilization, VRAM, temperature, power). Twin of hosts/esh-ml1.yaml: +# Docker-in-LXC, NVIDIA container toolkit with no-cgroups=true +# (playbooks/gpu-lxc.yaml). No extra filesystems: the root filesystem holds +# everything, models included. +services: + beszel-agent: + image: henrygd/beszel-agent-nvidia:0.18.7 + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: all + capabilities: [utility] diff --git a/stacks/dozzle/README.md b/stacks/dozzle/README.md index b03b7fc..0e32a8f 100644 --- a/stacks/dozzle/README.md +++ b/stacks/dozzle/README.md @@ -8,6 +8,7 @@ Container log viewer. One UI on **ana-docker** aggregates logs from every Docker - **esh-docker-vm** / **vm-esh-nas** (agents) — `10.0.50.45:7007`, `10.0.50.154:7007` - **irv-ml1** (agent) — `10.6.110.50:7007` (mesh address) - **esh-ml1** (agent, added 2026-09-25) — `10.0.50.80:7007`, pinned `v10.4.1` = the hub's version; compose dir `dozzle-agent`. The host needed an (empty) `traefik-net` network because the compose declares it external. +- **nh3-ml1** (agent, added 2026-09-25) — `10.100.50.80:7007`, same shape as esh-ml1 (`v10.4.1`, `dozzle-agent`, empty `traefik-net`). Hub `clients` went 7 → 8. - **nh3-docker** (agent, cross-site) — `10.100.50.40:7007`. ⚠ **Stopped by hand ~2026-04 (Exited 0) and left that way**; the hub logs a refused connection for it. Revive it or drop it from the list deliberately. ⚠ **The hub's agent list silently rots when a host moves.** Until 2026-09-25 it diff --git a/stacks/embed-rerank/README.md b/stacks/embed-rerank/README.md index 687c1a7..6f1d304 100644 --- a/stacks/embed-rerank/README.md +++ b/stacks/embed-rerank/README.md @@ -3,7 +3,10 @@ **The fleet's embedding + reranking service**, on **esh-ml1** (CT 110 on esh-pve, RTX 2000E Ada), served by **Hugging Face Text Embeddings Inference (TEI)**. Since 2026-09-25 it is the only backend behind the gateway names -`qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3`. +`qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3`. A second instance +runs on **nh3-ml1** (CT 109 on nh3-pve, the same card), parity-verified against +esh-ml1 on 2026-09-25 (`servers/nh3-ml1/README.md`). It is not in the gateway +yet; that is Prime's call. **TEI is the fleet's embed/rerank engine** (Prime, 2026-09-25). New embedding or reranking seats go on TEI, not vLLM. Why, and the measurements behind it: @@ -43,6 +46,9 @@ scripts/deploy-stack.sh esh-ml1 embed-rerank ssh esh-ml1 'cd /opt/docker/compose/embed-rerank && cp -n .env.example .env && docker compose config -q && docker compose up -d' ``` +nh3-ml1 is the same, plus `HOST_NAME=nh3-ml1` and `HOST_IP=10.100.50.80` in its +`.env` (they only feed the Homepage labels). + Host prerequisites (driver, LXC, docker, toolkit) are in [`servers/esh-ml1/README.md`](../../servers/esh-ml1/README.md). diff --git a/stacks/homepage/conf/docker.yaml b/stacks/homepage/conf/docker.yaml index 8afa0a5..a6318ad 100644 --- a/stacks/homepage/conf/docker.yaml +++ b/stacks/homepage/conf/docker.yaml @@ -37,6 +37,12 @@ esh-ml1-docker: host: 10.0.50.80 port: 2375 +# nh3-ml1 — CT 109 on nh3-pve: the second embed/rerank (TEI) backend. +# dockerd listens only on its own address (playbooks/gpu-lxc.yaml), 2026-09-25. +nh3-ml1-docker: + host: 10.100.50.80 + port: 2375 + # Example TLS socket (if/when a host moves off plaintext 2375): # ana-pfi-docker: # host: 10.250.50.70 diff --git a/stacks/uptimekuma/monitors.yaml b/stacks/uptimekuma/monitors.yaml index 1361cfe..572f86d 100644 --- a/stacks/uptimekuma/monitors.yaml +++ b/stacks/uptimekuma/monitors.yaml @@ -93,19 +93,28 @@ monitors: url: http://10.250.50.70:8200/api/v1/services # ---- fleet embed/rerank: the one EXCEPTION to "seats are OUT" ---- - # Since 2026-09-25 these are the SOLE backends behind the gateway's - # `qwen3-embedding` and `reranker` (TEI on esh-ml1, no failover until the second - # RTX 2000 arrives). They are not come-and-go seats: dead = Worldtree recall, - # nevermore clustering and Open WebUI RAG all fail. TEI's /health runs the - # backend, so a loaded-but-broken model reads DOWN, not UP. The reward seat on the - # same box stays OUT: it has no working consumer (stacks/reward-seat/README.md). + # Since 2026-09-25 these back the gateway's `qwen3-embedding` and `reranker` + # (TEI on esh-ml1; nh3-ml1 is the second RTX 2000, parity-verified the same day, + # gateway routing pending Prime). They are not come-and-go seats: dead = + # Worldtree recall, nevermore clustering and Open WebUI RAG all fail. TEI's + # /health runs the backend, so a loaded-but-broken model reads DOWN, not UP. The + # reward seat on esh-ml1 stays OUT: it has no working consumer + # (stacks/reward-seat/README.md). - name: Embed — Qwen3 0.6B (TEI, esh-ml1) url: http://10.0.50.80:8001/health - description: sole backend for gateway `qwen3-embedding` + description: gateway `qwen3-embedding` backend (esh-ml1) - name: Rerank — bge-v2-m3 (TEI, esh-ml1) url: http://10.0.50.80:8013/health - description: sole backend for gateway `reranker` + description: gateway `reranker` backend (esh-ml1) + + - name: Embed — Qwen3 0.6B (TEI, nh3-ml1) + url: http://10.100.50.80:8001/health + description: second `qwen3-embedding` backend (nh3-ml1) + + - name: Rerank — bge-v2-m3 (TEI, nh3-ml1) + url: http://10.100.50.80:8013/health + description: second `reranker` backend (nh3-ml1) - name: talk url: https://talk.nh3.phasefinal.com:8092/