memory: snapshot — nh3-pve shutting down for the NH3 site visit (post-boot checklist), nh3-ml1 blocked on Secure Boot, gx10 AC-restore validated
This commit is contained in:
@@ -0,0 +1,61 @@
|
||||
# `[2026-09-25]` nh3-ml1 — second embed/rerank GPU LXC on nh3-pve (DEFERRED: Secure Boot)
|
||||
|
||||
**Ask (Prime, ~1410 PT):** "stand up an nh3-ml1 that takes advantage of that GPU"
|
||||
(the RTX 2000E Ada installed in nh3-pve that morning, `01:00.0`).
|
||||
|
||||
## Decisions taken on Prime's behalf (he said "clear to proceed" at ~1418)
|
||||
|
||||
- **Purpose: the second TEI embed/rerank backend.** esh-ml1's README had been
|
||||
waiting on "the second RTX 2000" for failover. No reward-seat copy is planned,
|
||||
because that seat has no working consumer.
|
||||
- **Placement: CT 109, `10.100.50.80/24`, VLAN 50, gw `.1`, rootfs on `local-zfs`,
|
||||
6 cores / 16 GB / 80 GB, startup order 30.** The address came from the UDM
|
||||
(`nh3-servers` DHCP pool `.150–.249`; in use `.40 .42 .45 .46 .50 .60 .71 .90`).
|
||||
`.80` mirrors esh-ml1. nh3-pve had 40 of 62 GB RAM free.
|
||||
- **Not backed up**, like esh-ml1. nh3-pve's vzdump job is `all 1` (exclude 107),
|
||||
so `gpu-lxc.yaml` adds the CT to that job's `exclude` list. The guard and the
|
||||
step were tested against sample configs: one positive case, plus negatives for
|
||||
already excluded, a vmid-list job, a non-vzdump `all 1` stanza and a missing
|
||||
file.
|
||||
- **Playbooks made host-generic (`bc278d4`):** `esh-pve-nvidia-host` →
|
||||
`pve-nvidia-host`, which gained headers/dkms/build-essential, a nouveau
|
||||
blacklist and an unload that refuses if nouveau has bound a device.
|
||||
`esh-ml1-lxc` → `gpu-lxc`, whose host vars have **no defaults**; elway aborts on
|
||||
an undefined `{{ var }}` (tested). Per-host `--var` lines are in its header.
|
||||
- **`stacks/embed-rerank` Homepage labels** take `HOST_NAME` / `HOST_IP`. The
|
||||
defaults render byte-identical to esh-ml1's live labels, so nothing is
|
||||
recreated there.
|
||||
|
||||
## What happened
|
||||
|
||||
1. The first `pve-nvidia-host` run died on host-key verification for
|
||||
`nh3-pve.nh3.internal`. Every `when:` got ssh rc 255 and SKIPPED, and verify
|
||||
failed. I pinned the name after confirming its ed25519 key equals the known
|
||||
IP entry.
|
||||
2. The second run installed headers 6.8.12-11 + `proxmox-headers-6.8` + dkms +
|
||||
build-essential, blacklisted and unloaded nouveau (it had failed its probe
|
||||
with `gsp ctor failed: -2` and was bound to nothing), then **the DKMS module
|
||||
was refused at load: Secure Boot is ON** (`mokutil --sb-state` enabled;
|
||||
lockdown `integrity`; the only MOK is the Proxmox Secure Boot CA). esh-pve,
|
||||
the same MS-01 and BIOS, has Secure Boot OFF. The installer rolled back: no
|
||||
dkms entry, no `nvidia-smi`. All guests were unaffected.
|
||||
3. `7ddd116` added a pre-flight that refuses when Secure Boot is on without an
|
||||
enrolled `/var/lib/dkms/mok.pub`. Tested: it refuses on nh3-pve and passes on
|
||||
esh-pve.
|
||||
|
||||
## Open calls (Prime's)
|
||||
|
||||
- **Secure Boot:** (1) turn it off in the BIOS on the NH3 visit (recommended; one
|
||||
more toggle in the menu he is already in for IGFX, and it matches esh-pve);
|
||||
(2) keep it and enroll a DKMS MOK (prep remotely, then answer MokManager at
|
||||
boot; DKMS 3.x then signs rebuilds itself); (3) a VFIO VM (works without a
|
||||
visit, but reverses his 09-24 "LXC + host driver, not VFIO" rule, and the RTX is
|
||||
currently the boot VGA). Options 1 and 2 need the console at boot, which is
|
||||
blind until IGFX is set, so the visit carries both.
|
||||
- **Gateway routing** once parity passes: recommendation load-share. The two
|
||||
cards are identical, and the old "don't load-share" rule came from pairing
|
||||
esh-ml1 with the much faster fv-ml1.
|
||||
|
||||
## Resume
|
||||
|
||||
The Current-state checklist, step 9. Nothing host-side exists yet for CT 109.
|
||||
+50
-68
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-25 ~1336 PT (⭐⭐ nh3-pve back from the GPU install, checklist passed; ⚠ iGPU hidden by BIOS → NanoKVM blind, HELD to next NH3 visit (Prime); gx10 AC-restore VALIDATED. Today: TEI = fleet embed/rerank engine, esh-ml1 sole backend + reward seat moved there, fv-ml1 GPU 1 freed ~15 GB; esh-ml1 monitoring wired; homepage wedge cleared; NIC names pinned on nh3-pve. 34 old entries archived.)_
|
||||
_Last updated: 2026-09-25 ~1430 PT (⭐⭐ nh3-pve SHUT DOWN by Prime for the NH3 site visit — post-boot checklist in Current state. nh3-ml1 blocked on nh3-pve SECURE BOOT; Prime's SB call pending. NanoKVM blind until IGFX fix on the visit; gx10 AC-restore VALIDATED; GPU playbooks host-generic.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -115,74 +115,52 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-09-24 ~2245 PT._
|
||||
_As of 2026-09-25 ~1430 PT._
|
||||
|
||||
### ⭐⭐ nh3-pve GPU install — host back 2026-09-25 1116, checklist run 1122
|
||||
### ⭐⭐ nh3-pve SHUT DOWN 2026-09-25 ~1430 PT (Prime's order) for the NH3 site visit
|
||||
|
||||
**Result:** checklist items 1 and 3–6 pass, and nothing needed restarting. Guests,
|
||||
NFS, the post office (push), svos, hermes-gateway, Booth, talk, ttyd, zellij,
|
||||
backup timers, pbs-nh3 datastore, the mesh (irv-ml1 direct), the egress proxy and
|
||||
NH3 DNS are all fine. Open items:
|
||||
- ⚠ **iGPU gone → NanoKVM has no picture. Prime's call (1333, via Miranda): HOLD,
|
||||
and stay blind until the next NH3 visit.** No BIOS change and no reboot. Target:
|
||||
the NanoKVM moves to pfi-gx10, and nh3-pve gets its own vPro/AMT. ⚠ AMT KVM
|
||||
captures only the iGPU, so the IGFX BIOS fix has to happen on that same visit,
|
||||
before the NanoKVM leaves. The order is in `servers/nh3-pve/README.md`.
|
||||
- The uplink cable moved from `enp2s0f0np0` to `enp2s0f1np1`. It is harmless
|
||||
because both ports are bridge members. Recorded in the README.
|
||||
- ✅ **pfi-gx10 AC-restore patch VALIDATED.** Prime confirmed he replugged AC and
|
||||
it came up by itself at 1108:54; the README is updated.
|
||||
- Pre-existing and not caused by the reboot: `althing-pump-infra-hermes` and
|
||||
`althing-seat-daemon-jekyll` are both disabled (stopped 09-19 and 09-24). The
|
||||
`infra-hermes` handle shows `pull`.
|
||||
This session runs ON nh3-dev and dies with it. nh3-dev wipes /tmp at boot, so there
|
||||
is NO /tmp handoff after this. **Successor: work this list first.**
|
||||
|
||||
**nh3-ml1 (Prime, 2026-09-25 ~1410): IN PROGRESS.** CT 109 on nh3-pve at
|
||||
10.100.50.80 (VLAN 50, outside the UDM pool .150–.249), local-zfs, a second
|
||||
embed/rerank (TEI) backend on the new RTX. Playbooks are host-generic as of
|
||||
`bc278d4`. Prime cleared the driver install at ~1418. **It then failed: nh3-pve has
|
||||
SECURE BOOT ON** (esh-pve has it off), and the unsigned DKMS module was refused.
|
||||
The installer rolled back. The playbook now refuses on this up front. Unblocking
|
||||
needs the console at boot (MokManager enroll, or SB off in the BIOS), which ties
|
||||
it to the IGFX/NanoKVM visit, unless Prime picks a VFIO VM. Waiting on his call.
|
||||
After it: `gpu-lxc.yaml` with nh3 vars → embed-rerank with
|
||||
`HOST_NAME/HOST_IP` → parity vs esh-ml1 → monitoring and DNS → gateway routing
|
||||
(Prime's call).
|
||||
|
||||
The pre-shutdown record follows, kept for the next GPU visit:
|
||||
|
||||
Pre-flight done before shutdown:
|
||||
- NIC names pinned by MAC (a GPU renames every NIC on this MS-01), initramfs
|
||||
rebuilt and ESP synced.
|
||||
- AMT port `enp88s0` removed from vmbr0 in the file.
|
||||
- nh3-dev's Beszel agent no longer binds NAS shares.
|
||||
- Details in `servers/nh3-pve/README.md`.
|
||||
**The site-visit plan.** Prime has mini-DP→DP adapters, so a DisplayPort monitor on
|
||||
the RTX plus a USB keyboard covers all of it:
|
||||
1. BIOS: set Primary Display = IGFX. The Auto setting hid the iGPU, so the
|
||||
NanoKVM is blind.
|
||||
2. BIOS/Secure Boot: Prime's call is PENDING. (1) SB off (recommended; matches
|
||||
esh-pve), (2) keep SB and enroll a DKMS MOK via MokManager, or (3) a VFIO VM
|
||||
instead of the LXC.
|
||||
3. vPro: cable I226-LM `enp88s0`, provision MEBx, confirm AMT KVM shows the console.
|
||||
4. Only then move the NanoKVM to pfi-gx10.
|
||||
Detail and order: `servers/nh3-pve/README.md`.
|
||||
|
||||
**Post-boot checklist:**
|
||||
1. **Host on the network:** `ping 10.100.250.60`, then `ip -br link` — names
|
||||
unchanged, `enp2s0f0np0` UP inside vmbr0. If there's no network, open the
|
||||
NanoKVM console at **https://10.100.250.171**.
|
||||
2. **GPU seen:** `lspci | grep -i nvidia` → expect `01:00.0`, and the X710 moved to
|
||||
`03:00` while keeping its pinned name. The BIOS primary display must stay on the
|
||||
iGPU, or the NanoKVM goes black.
|
||||
3. **Guests:**
|
||||
- running: 100 nh3-docker1, 101 nh3-extdev, 102 nh3-dev, 105 pbs-nh3, CT 103
|
||||
nh3-wg, CT 106 nh3-headscale, CT 107 nh3-scale;
|
||||
- staying off (expected): 104 nh3-laser, 108 opnsense-lab.
|
||||
4. **NFS:** touch the automounts — nh3-docker `/mnt/compose` `/mnt/backup`, nh3-dev
|
||||
`/mnt/backup` `/mnt/smithy`.
|
||||
5. **Post office** reachable (`postbox status`, `mode: push`). Kuma "althing post
|
||||
office" green.
|
||||
6. **nh3-dev services:** svos :8770 + hermes-gateway (Miranda's channel), Booth
|
||||
:8090, ttyd/zellij seats, `dev-backup.timer`. Beszel shows nh3-dev /
|
||||
nh3-docker / pbs-nh3 up.
|
||||
7. **Claude sessions:** relaunching them is Prime's call.
|
||||
8. **pfi-gx10:** if Prime ran the AC-pull test on this visit, verify it came back
|
||||
by itself.
|
||||
9. **Next work:** the GPU's purpose is TBD with Prime; likely the second embed/rerank
|
||||
backend. Reuse `playbooks/pve-nvidia-host.yaml` / `gpu-lxc.yaml` (make
|
||||
them host-generic). nh3-pve runs kernel 6.8.12-11 with **no matching
|
||||
proxmox-headers installed**, on PVE 8.4.1 (Debian 12 template only). If the LM
|
||||
port was cabled, MEBx provisioning can be done through the NanoKVM.
|
||||
1. Network: `ping 10.100.250.60`; `ip -br link`. ⚠ The uplink is now
|
||||
`enp2s0f1np1` (the cable moved cages on 09-25 at 1116), not `enp2s0f0np0`. Both
|
||||
are in vmbr0.
|
||||
2. GPU: `lspci` shows `01:00.0` RTX 2000E. If IGFX was set, `00:02.0` is back with
|
||||
`boot_vga=1`. `mokutil --sb-state` shows whether Secure Boot is off.
|
||||
3. Guests running: 100 nh3-docker1, 101 nh3-extdev, 102 nh3-dev, 105 pbs-nh3,
|
||||
CT 103 nh3-wg, 106 nh3-headscale, 107 nh3-scale. Staying off: 104, 108.
|
||||
4. NFS automounts: nh3-docker `/mnt/compose` `/mnt/backup`; nh3-dev `/mnt/backup`
|
||||
`/mnt/smithy`; pbs-nh3 datastore.
|
||||
5. Post office `postbox status` shows push/reachable; Kuma is green.
|
||||
6. nh3-dev services: svos :8770, hermes-gateway (Miranda's channel), Booth :8090,
|
||||
talk, ttyd RO/RW, zellij-web, `dev-backup.timer`.
|
||||
7. Claude sessions: relaunching them is Prime's call.
|
||||
8. pfi-gx10: the AC-restore patch is validated, so a power cut brings it back on
|
||||
its own.
|
||||
9. Resume nh3-ml1, only once Secure Boot is resolved.
|
||||
`scripts/elway root@nh3-pve.nh3.internal --playbook playbooks/pve-nvidia-host.yaml`.
|
||||
Its pre-flight refuses while SB is on without an enrolled key. Then run
|
||||
`playbooks/gpu-lxc.yaml` with the nh3-ml1 `--var` line from its header, deploy
|
||||
`embed-rerank` with `.env` `HOST_NAME=nh3-ml1` `HOST_IP=10.100.50.80`, run a
|
||||
parity check against esh-ml1 (noise floor plus controls), then wire monitoring,
|
||||
DNS and `servers/nh3-ml1/README.md`. Gateway routing is Prime's call; the
|
||||
recommendation is load-share after parity.
|
||||
|
||||
**Left on nh3-pve by the failed 09-25 driver install** (harmless, and needed later):
|
||||
headers 6.8.12-11 plus the series meta, dkms, build-essential; nouveau
|
||||
blacklisted and unloaded; the `.run` staged in `/root/nvidia`.
|
||||
|
||||
### ⭐ esh-ml1 = the fleet's utility GPU host (2026-09-25): embed/rerank (TEI) + reward (vLLM)
|
||||
|
||||
@@ -220,15 +198,17 @@ in `servers/pfi-gx10/README.md`. → `persistent-memory.d/2026-09-24-gx10-ac-res
|
||||
|
||||
### Live threads
|
||||
|
||||
- **Claude sessions on nh3-dev** died in the NH3 outage (only infra-ops,
|
||||
infra-hermes, jekyll survived). Relaunch is Prime's call; not confirmed done.
|
||||
- **Claude sessions on nh3-dev:** Prime relaunched 3 after the 1116 boot. They die again
|
||||
with the ~1430 shutdown, and relaunching them is his call.
|
||||
- **Homepage `hermes-gateway seat` rename resolved (2026-09-25):** the operator's 2026-09-24 handle split is real (`postbox handles` records it; infra-hermes confirmed the seat daemon carries `hermes-gateway`). The rename was already live and is committed as `422a27c`.
|
||||
- **High Seat backup proof DONE 2026-09-25 0111:** infra-hermes reported and I verified that restic snapshot `cd113bef` (nh3-dev, 0100) contains `~/.high-seat/reports/grok.963ddd3f428b7fc7.jsonl`. Told him to retire the daily 0110 check unless svos-dev asked for an ongoing one.
|
||||
- ESH has a single outside route (esh-scale on esh-pve) — noted, untracked.
|
||||
- origin/main is current through `2118449`; only the snapshot commits after it are unpushed (2026-09-25). Push is Prime's call.
|
||||
- origin/main is current through `58390a4`. Everything after it is unpushed (`2bf418c`…`7ddd116` plus this snapshot, 2026-09-25). Pushing is Prime's call.
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-25]` **nh3-ml1 = CT 109 @ 10.100.50.80, second TEI embed/rerank backend on nh3-pve's RTX 2000E; GPU playbooks made host-generic — BUILD DEFERRED: blocked on nh3-pve Secure Boot until the NH3 visit; SB handling and gateway routing are Prime's calls.** Tracked: `7ddd116`, Current state. → `persistent-memory.d/2026-09-25-nh3-ml1-standup.md`
|
||||
- `[2026-09-25]` **nh3-pve OOB: HOLD blind until the next NH3 visit (Prime via Miranda, 1333).** Target: NanoKVM → pfi-gx10, and the MS-01 on its own vPro. The IGFX BIOS fix rides the same visit, since AMT KVM captures only the iGPU. Tracked: `servers/nh3-pve/README.md` (`2b49be4`, `cdd7605`).
|
||||
- `[2026-09-25]` **MS-01 foot-gun: a GPU in the PCIe slot renames every NIC** (the slot's root port takes bus 01, so the X710 goes `enp2s0f0np0`→`enp3s0f0np0`). On nh3-pve that means booting with no uplink. Pinned by MAC with `.link` files before the install, and the AMT port was pulled from vmbr0. → `servers/nh3-pve/README.md`, `playbooks/nh3-pve-pin-nic-names.yaml` (`2118449`)
|
||||
- `[2026-09-25]` **Direct ESH→esh-ml1 consumer path — PARKED** (no ESH-side callers in 7 days; consumers go through the gateway). Tracked at `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md`.
|
||||
- `[2026-09-25]` Created empty private repo `corviduo/norn` (git@gitea.phasefinal.com:corviduo/norn.git) as claude-bot, at brokkr-smithy-dev's relay of the operator's Norn ruling; the `norn-dev` handle is the operator's to declare.
|
||||
@@ -236,7 +216,7 @@ in `servers/pfi-gx10/README.md`. → `persistent-memory.d/2026-09-24-gx10-ac-res
|
||||
- `[2026-09-25]` **TEI adopted as the fleet embed/rerank engine (Prime); esh-ml1 is the sole gateway backend for `qwen3-embedding` + `reranker`; fv-ml1's vLLM embed/rerank seats retired (~6.1 GB freed); parakeet stays on fv-ml1.** → `persistent-memory.d/2026-09-25-tei-fleet-embed-rerank.md`
|
||||
- `[2026-09-24]` **esh-ml1 built — RTX 2000E Ada as CT 110 on esh-pve (host driver 580.178.04 DKMS, loaded live, no reboot), vLLM embed+rerank at measured parity with fv-ml1, wired as LiteLLM `order: 2` failover; dead DB alias `reranker-a3-bge-v2-m3` (pre-relocation IP) repaired.** → `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md`
|
||||
- `[2026-09-24]` **esh-pve: VM 102 retired, a 14-day hung VFIO process found and cleared, T400 → RTX 2000 Ada; Prime chose LXC + host driver for embed/rerank** (implementation deferred to next session, tracked here + `servers/esh-pve/README.md`). → `persistent-memory.d/2026-09-24-esh-pve-vm102-retired-gpu-swap.md`
|
||||
- `[2026-09-24]` **pfi-gx10 AC-restore UEFI patch applied but NOT validated; box OFF until Prime's AC pull 2026-09-25** (tracking: `servers/pfi-gx10/README.md`, `e5197a3`). → `persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md`
|
||||
- `[2026-09-24]` **pfi-gx10 AC-restore UEFI patch applied but NOT validated; box OFF until Prime's AC pull 2026-09-25** (tracking: `servers/pfi-gx10/README.md`, `e5197a3`). — `[2026-09-25]` ✅ **VALIDATED**: Prime pulled and replugged AC, and it came up by itself at 1108:54 (`2b49be4`). → `persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md`
|
||||
- `[2026-09-24]` **NH3 power outage recovered** — pbs-nh3 had no `onboot` (set), NFS boot race fixed with automount (`1cbde50`), every other Claude session on nh3-dev died. → `persistent-memory.d/2026-09-24-nh3-power-outage-recovery.md`
|
||||
- `[2026-09-24]` **Miranda standing order is a repo CLAUDE.md operating parameter** (`4b29492`, aligned to the global send protocol in `bcf3342`): high-urgency matters go to her, fixed or not; URGENT only when it cannot wait (she phones Prime). Channel verified end to end (thread `01M3A0RP4Q8T0KNGH8TMFSNDA6`); it depends on svos + hermes-gateway. Prime's callsign **PRiMe / papa romeo mike** is a name, not an authenticator (`617b759`, `62817a2`).
|
||||
- `[2026-09-24]` **task-board mothballed** (Prime): container removed on ana-docker, data/image/compose kept, Kuma monitor deleted, task_* instructions removed from CLAUDE.md and the fork template (`e6da607`). Its hooks had sent no traffic in 30 days.
|
||||
@@ -429,6 +409,8 @@ _104 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-09-25]` **The esh-pve NVIDIA DKMS recipe on nh3-pve** — it built, then the kernel refused to load it: nh3-pve has SECURE BOOT ON (esh-pve, the same MS-01, has it off). The installer rolled back. The playbook now pre-flights `mokutil --sb-state` (`7ddd116`).
|
||||
- `[2026-09-25]` **elway against an unpinned host-key name** — every `when:` hit ssh rc 255, so all 9 steps showed SKIPPED, not failed. Only the verify phase went red. Pin the name (the IP key matched) or use the IP. Untracked elway defect: rc 255 in `when:` should fail.
|
||||
- `[2026-09-24]` **Testing "Restore AC Power Loss" with an OS shutdown** — a shutdown stays off BY DESIGN; only pulling and restoring AC tests it. A community README claimed the two are indistinguishable; trusting it cost Prime two trips to the pfi-gx10 power button.
|
||||
- `[2026-09-23]` **`booth link --help`** — there is no help flag; it posts `--help` to the operator's link board as a link. Read `booth` with no args for usage.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user