# `[2026-09-25]` nh3-ml1 LIVE after the NH3 site visit

**Prime, ~1524 PT:** "secure boot disabled, amt enabled, card plugged." This is the
resume of `2026-09-25-nh3-ml1-standup.md`. nh3-pve came up at 1523.

## Post-boot checklist: all green

- Uplink is `enp2s0f1np1`. Guests 100, 101, 102, 105 and CTs 103, 106, 107 are up;
  104 and 108 are off as expected.
- NFS automounts are good on nh3-docker, nh3-dev and pbs-nh3. The post office is
  push/reachable. The 14 queued althing messages were all Beszel/Kuma "UP"
  recoveries, so nothing needed acting on.
- Secure Boot is disabled. `00:02.0` is back with `boot_vga=1` and i915 loaded.

## Build (elway, both playbooks idempotent)

1. `pve-nvidia-host.yaml`: the driver installed (DKMS, 36 s), 6/6 verify.
2. `gpu-lxc.yaml`: CT 109 built, then **verify 3 FAILED**. `docker run` died with
   `open sysctl net.ipv4.ip_unprivileged_port_start file: reopen fd 8: permission
   denied`.
   - The cause is the lxc-pve version, not runc. Both CTs run runc 1.5.1. nh3-pve
     (PVE 8.4.1) had lxc-pve **6.0.0-1**, and esh-pve (8.4.20) has **6.0.0-2**,
     whose changelog is Proxmox "fix #7006: do not restrict /proc or /sys if
     nesting is enabled".
   - **Decided autonomously:** upgrade that one package (a dry-run showed
     1 upgraded, 0 new, 0 removed) and `pct reboot 109`. GPU-in-docker then passed.
     ops-log recorded.
   - The playbook gained a first step that upgrades lxc-pve when it is below
     6.0.0-2, with a quoted name, because an unquoted `#7006` truncated it as a YAML
     comment. The re-run was 5/5 verify OK. The guard skips on esh-pve (tested).
3. `embed-rerank` deploy. `deploy-stack.sh --yes` was **denied by the auto-mode
   classifier as a blind apply**, so the pattern now is: preview with
   `echo n | deploy-stack.sh …` (it shows the diff and aborts), then `echo y |`.
   `.env` = example + `HOST_NAME=nh3-ml1`, `HOST_IP=10.100.50.80`. Healthy in about
   120 s including the image pull; 2.7 GB VRAM.

## Parity and speed (vs esh-ml1)

Script: session scratchpad, not committed. Corpus: 1,120 repo-doc paragraphs
plus 6 fixed texts, 2 runs per host, 50 queries.
- Embedding cosine esh↔nh3: median 0.999998, min 0.999993. The floors are esh
  0.999995 and nh3 0.999993.
- Negative control (different texts): median 0.50.
- overlap@10 = 1.000 both ways. Positive control, MRL-256: 0.684.
- Rerank, 100×20: top-1 1.00, max score diff 0.0014 (the esh-esh floor is
  0.0020). Positive control, a 4-word query: top-5 order 0.09, max diff 0.97.
- Speed on-box, 3 interleaved reps: short embed p50 6.6–6.7 vs 7.0 ms; bulk
  104–105 vs 106–108 passages/s; rerank-20 p50 156–159 vs 152–161 ms. The same.

## Monitoring and wiring

- Beszel: agent (`hosts/nh3-ml1.yaml`, same hub key as esh-ml1), system
  `1feeeq61g4mkqre`, 5 alerts, GPU samples present.
  - ⚠ zsh gotcha: `for a in "X 1 2"; do set -- $a` does NOT word-split in zsh, so
    the first batch of alert POSTs failed silently (empty fields). Use a function
    with explicit args.
- Kuma #29/#30 (the seed also refreshed the esh-ml1 descriptions). Homepage
  `nh3-ml1-docker` shows 2 cards with no restart. Dozzle agent added, and the hub
  `.env` backed up as `.env.bak-20260925-nh3ml1` before the edit.
- DNS `nh3-ml1.nh3.internal` on all 3 resolvers. `~/.ssh/config` `Host nh3-ml1`;
  `servers/nh3-ml1/` has ssh-target, system-details and README.

## Found along the way

- **Kernel 6.8.12-43 came from our own playbook.** The headers step's
  `proxmox-headers-6.8` install upgraded the `proxmox-kernel-6.8` meta
  (apt history 14:19:23). The visit reboot booted it. Documented in
  `pve-nvidia-host.yaml`.
- **btmtk Bluetooth oops on every -4x boot** (2/2 lines on each of 3 -43 boots,
  0 on the -11 boots; esh-pve -42 is the same). It taints `D`. Only the BT worker
  dies. Proposed fix: blacklist btusb on both PVE hosts. Not done.
- **AMT is not reachable.**
  - The LM port has a 1 Gb carrier (brief unbridged `ip link up` with IPv6
    disabled, then restored; ops-log recorded).
  - No UDM client or lease for `58:47:ca:76:96:0e`.
  - A /dev/tcp sweep of 16992/16993 over 10.100.{0,10,50,250}.0/24 found nothing.
    Its positive controls, :8390 and :8006, did answer.
  - The USW Pro 24's up ports (19, 22, 23, 26) are all other devices. So the cable
    is probably on nh3-sw1 (`10.100.250.2`), where SSH times out and no creds are
    held.
  - Asked Prime.

## Open (Prime's)

- Gateway routing for `qwen3-embedding` / `reranker`. Recommendation: load-share
  (identical cards, and a second site).
- AMT network settings and switch port. The NanoKVM stays on nh3-pve until AMT KVM
  shows the console.
- btusb blacklist on nh3-pve and esh-pve. Low priority.