62 lines
3.4 KiB
Markdown
62 lines
3.4 KiB
Markdown
# `[2026-09-25]` nh3-ml1 — second embed/rerank GPU LXC on nh3-pve (DEFERRED: Secure Boot)
|
||
|
||
**Ask (Prime, ~1410 PT):** "stand up an nh3-ml1 that takes advantage of that GPU"
|
||
(the RTX 2000E Ada installed in nh3-pve that morning, `01:00.0`).
|
||
|
||
## Decisions taken on Prime's behalf (he said "clear to proceed" at ~1418)
|
||
|
||
- **Purpose: the second TEI embed/rerank backend.** esh-ml1's README had been
|
||
waiting on "the second RTX 2000" for failover. No reward-seat copy is planned,
|
||
because that seat has no working consumer.
|
||
- **Placement: CT 109, `10.100.50.80/24`, VLAN 50, gw `.1`, rootfs on `local-zfs`,
|
||
6 cores / 16 GB / 80 GB, startup order 30.** The address came from the UDM
|
||
(`nh3-servers` DHCP pool `.150–.249`; in use `.40 .42 .45 .46 .50 .60 .71 .90`).
|
||
`.80` mirrors esh-ml1. nh3-pve had 40 of 62 GB RAM free.
|
||
- **Not backed up**, like esh-ml1. nh3-pve's vzdump job is `all 1` (exclude 107),
|
||
so `gpu-lxc.yaml` adds the CT to that job's `exclude` list. The guard and the
|
||
step were tested against sample configs: one positive case, plus negatives for
|
||
already excluded, a vmid-list job, a non-vzdump `all 1` stanza and a missing
|
||
file.
|
||
- **Playbooks made host-generic (`bc278d4`):** `esh-pve-nvidia-host` →
|
||
`pve-nvidia-host`, which gained headers/dkms/build-essential, a nouveau
|
||
blacklist and an unload that refuses if nouveau has bound a device.
|
||
`esh-ml1-lxc` → `gpu-lxc`, whose host vars have **no defaults**; elway aborts on
|
||
an undefined `{{ var }}` (tested). Per-host `--var` lines are in its header.
|
||
- **`stacks/embed-rerank` Homepage labels** take `HOST_NAME` / `HOST_IP`. The
|
||
defaults render byte-identical to esh-ml1's live labels, so nothing is
|
||
recreated there.
|
||
|
||
## What happened
|
||
|
||
1. The first `pve-nvidia-host` run died on host-key verification for
|
||
`nh3-pve.nh3.internal`. Every `when:` got ssh rc 255 and SKIPPED, and verify
|
||
failed. I pinned the name after confirming its ed25519 key equals the known
|
||
IP entry.
|
||
2. The second run installed headers 6.8.12-11 + `proxmox-headers-6.8` + dkms +
|
||
build-essential, blacklisted and unloaded nouveau (it had failed its probe
|
||
with `gsp ctor failed: -2` and was bound to nothing), then **the DKMS module
|
||
was refused at load: Secure Boot is ON** (`mokutil --sb-state` enabled;
|
||
lockdown `integrity`; the only MOK is the Proxmox Secure Boot CA). esh-pve,
|
||
the same MS-01 and BIOS, has Secure Boot OFF. The installer rolled back: no
|
||
dkms entry, no `nvidia-smi`. All guests were unaffected.
|
||
3. `7ddd116` added a pre-flight that refuses when Secure Boot is on without an
|
||
enrolled `/var/lib/dkms/mok.pub`. Tested: it refuses on nh3-pve and passes on
|
||
esh-pve.
|
||
|
||
## Open calls (Prime's)
|
||
|
||
- **Secure Boot:** (1) turn it off in the BIOS on the NH3 visit (recommended; one
|
||
more toggle in the menu he is already in for IGFX, and it matches esh-pve);
|
||
(2) keep it and enroll a DKMS MOK (prep remotely, then answer MokManager at
|
||
boot; DKMS 3.x then signs rebuilds itself); (3) a VFIO VM (works without a
|
||
visit, but reverses his 09-24 "LXC + host driver, not VFIO" rule, and the RTX is
|
||
currently the boot VGA). Options 1 and 2 need the console at boot, which is
|
||
blind until IGFX is set, so the visit carries both.
|
||
- **Gateway routing** once parity passes: recommendation load-share. The two
|
||
cards are identical, and the old "don't load-share" rule came from pairing
|
||
esh-ml1 with the much faster fv-ml1.
|
||
|
||
## Resume
|
||
|
||
The Current-state checklist, step 9. Nothing host-side exists yet for CT 109.
|