vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(nh3-ml1): second TEI embed/rerank backend live on nh3-pve; parity-verified vs esh-ml1

Browse Source 
NH3 site visit done: Secure Boot off, iGPU restored as boot VGA, AMT port cabled.

- nh3-pve: NVIDIA 580.178.04 (DKMS, open modules) via pve-nvidia-host.yaml.
- nh3-ml1 = CT 109 @ 10.100.50.80 via gpu-lxc.yaml; embed-rerank (TEI 1.9.4)
  deployed with HOST_NAME/HOST_IP labels.
- Parity vs esh-ml1 (1,126 texts, 2 runs/host, controls): embed cosine min
  0.999993 = own noise floor; overlap@10 1.000 vs MRL-256 positive control
  0.684; rerank top-1 1.00, max diff 0.0014 vs floor 0.0020. On-box speed
  identical within rep spread.
- gpu-lxc.yaml: first step upgrades lxc-pve to >= 6.0.0-2 (Proxmox fix #7006).
  With 6.0.0-1 every docker run in a nesting CT failed on runc 1.5's sysctl
  reopen; applied on nh3-pve (one package).
- pve-nvidia-host.yaml: document that the headers meta drags in the newest
  kernel (nh3-pve went 6.8.12-11 -> -43 at the next reboot).
- Monitoring: Beszel NVIDIA agent + 5 alerts, Kuma #29/#30, Homepage
  nh3-ml1-docker, Dozzle agent (hub 8 clients). DNS nh3-ml1.nh3.internal.
- nh3-pve README: SB/IGFX/driver/kernel state, btmtk oops on -4x kernels,
  AMT cabled but unreachable on the network.

Gateway routing to nh3-ml1 is not changed.
This commit is contained in:
vh
2026-09-25 15:55:33 -07:00
parent 6fa8213c20
commit 5960526c3f
17 changed files with 815 additions and 112 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+36 -49
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-09-25 ~1430 PT (⭐⭐ nh3-pve SHUT DOWN by Prime for the NH3 site visit — post-boot checklist in Current state. nh3-ml1 blocked on nh3-pve SECURE BOOT; Prime's SB call pending. NanoKVM blind until IGFX fix on the visit; gx10 AC-restore VALIDATED; GPU playbooks host-generic.)_
_Last updated: 2026-09-25 ~1555 PT (⭐⭐ NH3 visit DONE: SB off, IGFX restored, AMT cabled but NOT on the network. nh3-ml1 LIVE + parity-verified + monitored; gateway routing = Prime's call. lxc-pve fix; kernel -43 btmtk oops noted. gx10 AC-restore VALIDATED.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -115,57 +115,44 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight
_As of 2026-09-25 ~1430 PT._
_As of 2026-09-25 ~1555 PT._
### ⭐⭐ nh3-pve SHUT DOWN 2026-09-25 ~1430 PT (Prime's order) for the NH3 site visit
### ⭐⭐ NH3 site visit DONE; nh3-pve back up 1523; nh3-ml1 LIVE (not yet in the gateway)
This session runs ON nh3-dev and dies with it. nh3-dev wipes /tmp at boot, so there
is NO /tmp handoff after this. **Successor: work this list first.**
**The site-visit plan.** Prime has mini-DP→DP adapters, so a DisplayPort monitor on
the RTX plus a USB keyboard covers all of it:
1. BIOS: set Primary Display = IGFX. The Auto setting hid the iGPU, so the
NanoKVM is blind.
2. BIOS/Secure Boot: Prime's call is PENDING. (1) SB off (recommended; matches
esh-pve), (2) keep SB and enroll a DKMS MOK via MokManager, or (3) a VFIO VM
instead of the LXC.
3. vPro: cable I226-LM `enp88s0`, provision MEBx, confirm AMT KVM shows the console.
4. Only then move the NanoKVM to pfi-gx10.
Detail and order: `servers/nh3-pve/README.md`.
**Post-boot checklist:**
1. Network: `ping 10.100.250.60`; `ip -br link`. ⚠ The uplink is now
`enp2s0f1np1` (the cable moved cages on 09-25 at 1116), not `enp2s0f0np0`. Both
are in vmbr0.
2. GPU: `lspci` shows `01:00.0` RTX 2000E. If IGFX was set, `00:02.0` is back with
`boot_vga=1`. `mokutil --sb-state` shows whether Secure Boot is off.
3. Guests running: 100 nh3-docker1, 101 nh3-extdev, 102 nh3-dev, 105 pbs-nh3,
CT 103 nh3-wg, 106 nh3-headscale, 107 nh3-scale. Staying off: 104, 108.
4. NFS automounts: nh3-docker `/mnt/compose` `/mnt/backup`; nh3-dev `/mnt/backup`
`/mnt/smithy`; pbs-nh3 datastore.
5. Post office `postbox status` shows push/reachable; Kuma is green.
6. nh3-dev services: svos :8770, hermes-gateway (Miranda's channel), Booth :8090,
talk, ttyd RO/RW, zellij-web, `dev-backup.timer`.
7. Claude sessions: relaunching them is Prime's call.
8. pfi-gx10: the AC-restore patch is validated, so a power cut brings it back on
its own.
9. Resume nh3-ml1, only once Secure Boot is resolved.
`scripts/elway root@nh3-pve.nh3.internal --playbook playbooks/pve-nvidia-host.yaml`.
Its pre-flight refuses while SB is on without an enrolled key. Then run
`playbooks/gpu-lxc.yaml` with the nh3-ml1 `--var` line from its header, deploy
`embed-rerank` with `.env` `HOST_NAME=nh3-ml1` `HOST_IP=10.100.50.80`, run a
parity check against esh-ml1 (noise floor plus controls), then wire monitoring,
DNS and `servers/nh3-ml1/README.md`. Gateway routing is Prime's call; the
recommendation is load-share after parity.
**Left on nh3-pve by the failed 09-25 driver install** (harmless, and needed later):
headers 6.8.12-11 plus the series meta, dkms, build-essential; nouveau
blacklisted and unloaded; the `.run` staged in `/root/nvidia`.
Prime, after the visit: "secure boot disabled, amt enabled, card plugged." The
post-boot checklist is all green: the uplink is on `enp2s0f1np1`, all 7 onboot
guests are up, NFS automounts, the post office is push/reachable, and svos,
hermes-gateway, Booth, ttyd, zellij-web and dev-backup are running.
- **Secure Boot OFF. IGFX restored** (`00:02.0` `boot_vga=1`), so the NanoKVM should
have video again (not checked). NVIDIA 580.178.04 DKMS is on the host.
- **nh3-ml1 = CT 109 @ 10.100.50.80, TEI embed+rerank, parity-verified vs esh-ml1:
the two hosts cannot be told apart and run at the same speed.** Monitoring is
wired: Beszel `1feeeq61g4mkqre` with 5 alerts, Kuma #29/#30, Homepage cards,
Dozzle agent (hub 7→8 clients). DNS `nh3-ml1.nh3.internal` is set.
`servers/nh3-ml1/README.md` exists.
- ⏳ **Prime's call: gateway routing.** Recommendation: load-share
`qwen3-embedding` + `reranker` across esh-ml1 and nh3-ml1.
- ⚠ **AMT is cabled (1 Gb link) and reported enabled, but not on the network.** No
UDM lease for `…:96:0e`, and no 16992/16993 on the four NH3 /24s (the sweep was
positive-controlled). It is not on the USW Pro 24, so probably nh3-sw1. Asked
Prime for the MEBx network settings and which switch port. **The NanoKVM stays
on nh3-pve until AMT KVM is confirmed.**
- Found and fixed: **lxc-pve 6.0.0-1 broke Docker in the CT** (runc 1.5 sysctl
reopen denied). Upgraded that one package to 6.0.0-2 (Proxmox fix #7006), and
`gpu-lxc.yaml` now does this itself.
- Found, NOT fixed: the headers install on 09-25 pulled kernel **6.8.12-43**, which
booted at the visit. It oopses in Bluetooth (`btmtk`) on every boot, and so does
esh-pve on -42. Benign so far. The proposed fix is to blacklist `btusb` on both
PVE hosts, effective at the next boot.
- Not yet proven: nh3-ml1 coming back by itself after an nh3-pve reboot.
`nvidia-persistenced` is ordered before `pve-guests`, but only the config was
checked; nothing has been rebooted.
→ `persistent-memory.d/2026-09-25-nh3-ml1-live.md`
### ⭐ esh-ml1 = the fleet's utility GPU host (2026-09-25): embed/rerank (TEI) + reward (vLLM)
TEI is the fleet embed/rerank engine (Prime) and esh-ml1 is the SOLE backend
(⚠ no failover until the second RTX 2000 arrives). The Skywork reward seat moved
TEI is the fleet embed/rerank engine (Prime) and esh-ml1 is the SOLE gateway
backend (nh3-ml1 is live and parity-verified; routing it in is Prime's call). The Skywork reward seat moved
there too; the audit found nothing supersedes it. fv-ml1 GPU 1 freed ~15 GB
across the day; only `vllm-coder` is left in `stacks/vllm`, and **parakeet stays**
on fv-ml1. Monitoring is wired: Beszel with GPU and 5 alerts, Kuma #27/#28,
@@ -198,8 +185,7 @@ in `servers/pfi-gx10/README.md`. → `persistent-memory.d/2026-09-24-gx10-ac-res
### Live threads
- **Claude sessions on nh3-dev:** Prime relaunched 3 after the 1116 boot. They die again
with the ~1430 shutdown, and relaunching them is his call.
- **Claude sessions on nh3-dev:** the ~1430 shutdown killed them again; relaunching is Prime's call.
- **Homepage `hermes-gateway seat` rename resolved (2026-09-25):** the operator's 2026-09-24 handle split is real (`postbox handles` records it; infra-hermes confirmed the seat daemon carries `hermes-gateway`). The rename was already live and is committed as `422a27c`.
- **High Seat backup proof DONE 2026-09-25 0111:** infra-hermes reported and I verified that restic snapshot `cd113bef` (nh3-dev, 0100) contains `~/.high-seat/reports/grok.963ddd3f428b7fc7.jsonl`. Told him to retire the daily 0110 check unless svos-dev asked for an ongoing one.
- ESH has a single outside route (esh-scale on esh-pve) — noted, untracked.
@@ -207,6 +193,7 @@ in `servers/pfi-gx10/README.md`. → `persistent-memory.d/2026-09-24-gx10-ac-res
## Recent decisions
- `[2026-09-25]` **nh3-ml1 LIVE after the NH3 visit (SB off, IGFX restored): driver + CT 109 + TEI, parity vs esh-ml1 indistinguishable (cos min 0.999993 = own floor; overlap@10 1.000 vs MRL-256 control 0.684), same speed; monitoring/DNS wired. Autonomous: lxc-pve 6.0.0-2 upgrade on nh3-pve (Docker-in-LXC fix #7006). Gateway routing + AMT follow-up are Prime's.** → `persistent-memory.d/2026-09-25-nh3-ml1-live.md`
- `[2026-09-25]` **nh3-ml1 = CT 109 @ 10.100.50.80, second TEI embed/rerank backend on nh3-pve's RTX 2000E; GPU playbooks made host-generic — BUILD DEFERRED: blocked on nh3-pve Secure Boot until the NH3 visit; SB handling and gateway routing are Prime's calls.** Tracked: `7ddd116`, Current state. → `persistent-memory.d/2026-09-25-nh3-ml1-standup.md`
- `[2026-09-25]` **nh3-pve OOB: HOLD blind until the next NH3 visit (Prime via Miranda, 1333).** Target: NanoKVM → pfi-gx10, and the MS-01 on its own vPro. The IGFX BIOS fix rides the same visit, since AMT KVM captures only the iGPU. Tracked: `servers/nh3-pve/README.md` (`2b49be4`, `cdd7605`).
- `[2026-09-25]` **MS-01 foot-gun: a GPU in the PCIe slot renames every NIC** (the slot's root port takes bus 01, so the X710 goes `enp2s0f0np0`→`enp3s0f0np0`). On nh3-pve that means booting with no uplink. Pinned by MAC with `.link` files before the install, and the AMT port was pulled from vmbr0. → `servers/nh3-pve/README.md`, `playbooks/nh3-pve-pin-nic-names.yaml` (`2118449`)