esh-pfi-infrastructure/persistent-memory.d/2026-09-25-nh3-ml1-live.md

[2026-09-25] nh3-ml1 LIVE after the NH3 site visit

Prime, ~1524 PT: "secure boot disabled, amt enabled, card plugged." This is the resume of 2026-09-25-nh3-ml1-standup.md. nh3-pve came up at 1523.

Post-boot checklist: all green

• Uplink is enp2s0f1np1. Guests 100, 101, 102, 105 and CTs 103, 106, 107 are up; 104 and 108 are off as expected.
• NFS automounts are good on nh3-docker, nh3-dev and pbs-nh3. The post office is push/reachable. The 14 queued althing messages were all Beszel/Kuma "UP" recoveries, so nothing needed acting on.
• Secure Boot is disabled. 00:02.0 is back with boot_vga=1 and i915 loaded.

Build (elway, both playbooks idempotent)

1. pve-nvidia-host.yaml: the driver installed (DKMS, 36 s), 6/6 verify.
2. gpu-lxc.yaml: CT 109 built, then verify 3 FAILED. docker run died with open sysctl net.ipv4.ip_unprivileged_port_start file: reopen fd 8: permission denied.
   • The cause is the lxc-pve version, not runc. Both CTs run runc 1.5.1. nh3-pve (PVE 8.4.1) had lxc-pve 6.0.0-1, and esh-pve (8.4.20) has 6.0.0-2, whose changelog is Proxmox "fix #7006: do not restrict /proc or /sys if nesting is enabled".
   • Decided autonomously: upgrade that one package (a dry-run showed 1 upgraded, 0 new, 0 removed) and pct reboot 109. GPU-in-docker then passed. ops-log recorded.
   • The playbook gained a first step that upgrades lxc-pve when it is below 6.0.0-2, with a quoted name, because an unquoted #7006 truncated it as a YAML comment. The re-run was 5/5 verify OK. The guard skips on esh-pve (tested).
3. embed-rerank deploy. deploy-stack.sh --yes was denied by the auto-mode classifier as a blind apply, so the pattern now is: preview with echo n | deploy-stack.sh … (it shows the diff and aborts), then echo y |. .env = example + HOST_NAME=nh3-ml1, HOST_IP=10.100.50.80. Healthy in about 120 s including the image pull; 2.7 GB VRAM.

Parity and speed (vs esh-ml1)

Script: session scratchpad, not committed. Corpus: 1,120 repo-doc paragraphs plus 6 fixed texts, 2 runs per host, 50 queries.

• Embedding cosine esh↔nh3: median 0.999998, min 0.999993. The floors are esh 0.999995 and nh3 0.999993.
• Negative control (different texts): median 0.50.
• overlap@10 = 1.000 both ways. Positive control, MRL-256: 0.684.
• Rerank, 100×20: top-1 1.00, max score diff 0.0014 (the esh-esh floor is 0.0020). Positive control, a 4-word query: top-5 order 0.09, max diff 0.97.
• Speed on-box, 3 interleaved reps: short embed p50 6.6–6.7 vs 7.0 ms; bulk 104–105 vs 106–108 passages/s; rerank-20 p50 156–159 vs 152–161 ms. The same.

Monitoring and wiring

• Beszel: agent (hosts/nh3-ml1.yaml, same hub key as esh-ml1), system 1feeeq61g4mkqre, 5 alerts, GPU samples present.
  • ⚠ zsh gotcha: for a in "X 1 2"; do set -- $a does NOT word-split in zsh, so the first batch of alert POSTs failed silently (empty fields). Use a function with explicit args.
• Kuma #29/#30 (the seed also refreshed the esh-ml1 descriptions). Homepage nh3-ml1-docker shows 2 cards with no restart. Dozzle agent added, and the hub .env backed up as .env.bak-20260925-nh3ml1 before the edit.
• DNS nh3-ml1.nh3.internal on all 3 resolvers. ~/.ssh/config Host nh3-ml1; servers/nh3-ml1/ has ssh-target, system-details and README.

Found along the way

• Kernel 6.8.12-43 came from our own playbook. The headers step's proxmox-headers-6.8 install upgraded the proxmox-kernel-6.8 meta (apt history 14:19:23). The visit reboot booted it. Documented in pve-nvidia-host.yaml.
• btmtk Bluetooth oops on every -4x boot (2/2 lines on each of 3 -43 boots, 0 on the -11 boots; esh-pve -42 is the same). It taints D. Only the BT worker dies. Proposed fix: blacklist btusb on both PVE hosts. Not done.
• AMT is not reachable.
  • The LM port has a 1 Gb carrier (brief unbridged ip link up with IPv6 disabled, then restored; ops-log recorded).
  • No UDM client or lease for 58:47:ca:76:96:0e.
  • A /dev/tcp sweep of 16992/16993 over 10.100.{0,10,50,250}.0/24 found nothing. Its positive controls, :8390 and :8006, did answer.
  • The USW Pro 24's up ports (19, 22, 23, 26) are all other devices. So the cable is probably on nh3-sw1 (10.100.250.2), where SSH times out and no creds are held.
  • Asked Prime.

Open (Prime's)

• Gateway routing for qwen3-embedding / reranker. Recommendation: load-share (identical cards, and a second site).
• AMT network settings and switch port. The NanoKVM stays on nh3-pve until AMT KVM shows the console.
• btusb blacklist on nh3-pve and esh-pve. Low priority.