diff --git a/persistent-memory.md b/persistent-memory.md index 8ca82e2..f6b920d 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -117,6 +117,46 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-09-24 ~2245 PT._ +### ⭐⭐ IN PROGRESS 2026-09-25 ~1100 PT: nh3-pve shut down for a GPU install (Prime on site) + +This session runs ON nh3-dev and dies with it. **Successor: work this list +first.** (No /tmp handoff: nh3-dev wipes /tmp at boot.) + +Pre-flight done before shutdown: +- NIC names pinned by MAC (a GPU renames every NIC on this MS-01), initramfs + rebuilt and ESP synced. +- AMT port `enp88s0` removed from vmbr0 in the file. +- nh3-dev's Beszel agent no longer binds NAS shares. +- Details in `servers/nh3-pve/README.md`. + +**Post-boot checklist:** +1. **Host on the network:** `ping 10.100.250.60`, then `ip -br link` — names + unchanged, `enp2s0f0np0` UP inside vmbr0. If there's no network, open the + NanoKVM console at **https://10.100.250.171**. +2. **GPU seen:** `lspci | grep -i nvidia` → expect `01:00.0`, and the X710 moved to + `03:00` while keeping its pinned name. The BIOS primary display must stay on the + iGPU, or the NanoKVM goes black. +3. **Guests:** + - running: 100 nh3-docker1, 101 nh3-extdev, 102 nh3-dev, 105 pbs-nh3, CT 103 + nh3-wg, CT 106 nh3-headscale, CT 107 nh3-scale; + - staying off (expected): 104 nh3-laser, 108 opnsense-lab. +4. **NFS:** touch the automounts — nh3-docker `/mnt/compose` `/mnt/backup`, nh3-dev + `/mnt/backup` `/mnt/smithy`. +5. **Post office** reachable (`postbox status`, `mode: push`). Kuma "althing post + office" green. +6. **nh3-dev services:** svos :8770 + hermes-gateway (Miranda's channel), Booth + :8090, ttyd/zellij seats, `dev-backup.timer`. Beszel shows nh3-dev / + nh3-docker / pbs-nh3 up. +7. **Claude sessions:** relaunching them is Prime's call. +8. **pfi-gx10:** if Prime ran the AC-pull test on this visit, verify it came back + by itself. +9. **Next work:** the GPU's purpose is TBD with Prime; likely the second embed/rerank + backend. Reuse `playbooks/esh-pve-nvidia-host.yaml` / `esh-ml1-lxc.yaml` (make + them host-generic). nh3-pve runs kernel 6.8.12-11 with **no matching + proxmox-headers installed**, on PVE 8.4.1 (Debian 12 template only). If the LM + port was cabled, MEBx provisioning can be done through the NanoKVM. + + ### ⭐ esh-ml1 = the fleet's utility GPU host (2026-09-25): embed/rerank (TEI) + reward (vLLM) TEI is the fleet embed/rerank engine (Prime) and esh-ml1 is the SOLE backend diff --git a/playbooks/nh3-pve-pin-nic-names.yaml b/playbooks/nh3-pve-pin-nic-names.yaml new file mode 100644 index 0000000..41bea4c --- /dev/null +++ b/playbooks/nh3-pve-pin-nic-names.yaml @@ -0,0 +1,91 @@ +# nh3-pve — pin NIC names by MAC BEFORE a GPU goes into the PCIe slot, and take +# the AMT-capable port out of vmbr0. +# +# WHY (measured 2026-09-25, not assumed): nh3-pve and esh-pve are the same box — +# Minisforum MS-01 ("Venus Series"), BIOS AHWSA.1.17. On esh-pve, which has a GPU, +# the x16 slot's root port 00:01.0 takes PCI bus 01 and every device behind it +# moves down one bus: X710 at 03:00, I226-V 58, I226-LM 59. On nh3-pve (no GPU) +# 00:01.0 does not exist: X710 02:00, I226-V 57, I226-LM 58. Predictable NIC names +# encode the bus, so fitting a GPU renames them: +# enp2s0f0np0 -> enp3s0f0np0 (the ONLY connected uplink, SFP+) +# enp87s0 -> enp88s0, enp88s0 -> enp89s0, wlp89s0 -> wlp90s0 +# vmbr0's bridge-ports name the old spellings, so the host would boot with NO +# uplink: the NH3 post office, nh3-dev, DNS and the mesh subnet router all go dark. +# The fix is a systemd .link file per NIC that matches the MAC (which does not +# change) and keeps today's name. Nothing changes on the current boot. +# +# AMT: the vPro port is the I226-LM (enp88s0, MAC ...:0e). vmbr0 bridges ALL +# four NICs with STP off, so cabling the LM port while the SFP+ uplink is also +# connected would put two paths into the same L2 — a switching loop that would +# take down the site LAN. enp88s0 is removed from bridge-ports, in the FILE only: +# it takes effect on the next boot (the GPU install), with no live ifreload on the +# site's hypervisor. enp87s0 stays first in the list so vmbr0 keeps its MAC. +# +# Run: scripts/elway root@nh3-pve --playbook playbooks/nh3-pve-pin-nic-names.yaml + +vars: + stage_dir: /root/nic-pin-2026-09-25 + +steps: + - name: Back up the network config + shell: | + mkdir -p {{ stage_dir }} + cp -p /etc/network/interfaces {{ stage_dir }}/interfaces.before + creates: "{{ stage_dir }}/interfaces.before" + + - name: Pin each NIC's current name to its MAC (systemd .link) + shell: | + set -e + pin() { # name mac + cat > /etc/systemd/network/10-pin-$1.link < /etc/systemd/network/10-pin-wlp89s0.link <<'EOF' + # See eshpfi playbooks/nh3-pve-pin-nic-names.yaml + [Match] + MACAddress=4c:50:dd:6c:0d:f9 + [Link] + Name=wlp89s0 + EOF + when: "! test -f /etc/systemd/network/10-pin-enp2s0f0np0.link" + + - name: Take the AMT port (enp88s0) out of vmbr0 in the file (next boot) + shell: sed -i 's/^\(\s*bridge-ports\) enp87s0 enp88s0 enp2s0f0np0 enp2s0f1np1$/\1 enp87s0 enp2s0f0np0 enp2s0f1np1/' /etc/network/interfaces + when: "grep -qE '^\\s*bridge-ports enp87s0 enp88s0 enp2s0f0np0 enp2s0f1np1$' /etc/network/interfaces" + + - name: Rebuild the initramfs so the .link files apply in early boot too + # proxmox-boot-tool's post-update hook syncs the ESP. + shell: update-initramfs -u -k all + when: "! lsinitramfs /boot/initrd.img-$(uname -r) | grep -q '10-pin-enp2s0f0np0.link'" + +verify: + - name: interfaces file still parses (no live reload) + shell: ifreload -a -s + changed_when: "false" + + - name: bridge-ports no longer include the AMT port + shell: "grep -qE '^\\s*bridge-ports enp87s0 enp2s0f0np0 enp2s0f1np1$' /etc/network/interfaces" + changed_when: "false" + + - name: udev would give every wired NIC its pinned name via OUR .link file + shell: | + for n in enp2s0f0np0 enp2s0f1np1 enp87s0 enp88s0; do + out=$(udevadm test-builtin net_setup_link /sys/class/net/$n 2>&1) + echo "$out" | grep -q "10-pin-$n.link" || { echo "$n: not matched by 10-pin-$n.link"; exit 1; } + done + changed_when: "false" + + - name: .link files are inside the running kernel's initramfs + shell: lsinitramfs /boot/initrd.img-$(uname -r) | grep -q '10-pin-enp2s0f0np0.link' + changed_when: "false" diff --git a/servers/nh3-pve/README.md b/servers/nh3-pve/README.md index f99894e..fed832a 100644 --- a/servers/nh3-pve/README.md +++ b/servers/nh3-pve/README.md @@ -43,6 +43,39 @@ on), which is the root path into pbs-nh3; infra-ops is not provisioned there. **pfi-gx10 (`10.100.50.60`) is bare metal on this site, not a guest here**, and did not power back on by itself. +## Hardware identity, OOB, and the GPU-slot NIC-rename trap (2026-09-25) + +- **Box:** Minisforum MS-01 ("Venus Series", BIOS `AHWSA.1.17`): the same model + and BIOS as esh-pve. i9-13900H, X710 2×SFP+, I226-V + **I226-LM (vPro/AMT)**, + one PCIe x16 slot (x8 electrical, low-profile, single-slot, slot power only). +- **Uplink:** only `enp2s0f0np0` (SFP+ 10G) has a cable. The bridge carries the + I226-V's MAC `…:96:0d` because it is the first port listed. +- **AMT: NOT wired.** The AMT-capable I226-LM (`enp88s0`, MAC `58:47:ca:76:96:0e`) + has no cable. The ME is present (`/dev/mei0`, "AMT SOL Redirection" 00:16.3), but + MEBx provisioning status is unknown. To wire it: cable the LM port, then at boot + press Ctrl+P → set the MEBx password, enable manageability, set network (static + or DHCP), KVM on, User Opt-in = None, activate network access. MEBx can be driven + remotely through the NanoKVM below. +- **Console OOB exists: a Sipeed NanoKVM** is attached (USB `3346:1009` on the host; + web UI **`https://10.100.250.171`**, switch port 23, nh3-mgmt). It gives video and + keyboard, so BIOS, MEBx and a host that booted without network are all reachable + remotely. Power control through it is unverified (the MS-01 has no standard ATX + front-panel header). ⚠ It captures the **iGPU's HDMI**, and RTX-class cards output + mini-DP only, so keep the BIOS primary display on the iGPU after a GPU install. +- ⚠ **A GPU in the slot RENAMES every NIC.** With a card present the slot's root + port 00:01.0 takes PCI bus 01 and everything behind it moves down a bus (measured + on esh-pve): X710 02→03, I226-V 57→58, I226-LM 58→59, Wi-Fi 59→5a. The names + would become `enp3s0f0np0`, `enp88s0`, `enp89s0`, `wlp90s0`. vmbr0 would then boot + with no uplink and take the whole NH3 site's VMs off the network. + **Pinned 2026-09-25** by MAC with `/etc/systemd/network/10-pin-*.link`, also baked + into every initramfs and synced to the ESP (`playbooks/nh3-pve-pin-nic-names.yaml`, + verified with `udevadm test-builtin net_setup_link`). If the host ever boots + without network anyway: open the NanoKVM console, run `ip -br link`, and compare + names against `/etc/network/interfaces` (backup in `/root/nic-pin-2026-09-25/`). +- **`enp88s0` (the AMT port) is no longer a vmbr0 bridge port** (file edited + 2026-09-25, effective next boot). STP is off, so bridging a second cabled uplink + into the same L2 would loop the site LAN. + ## Refresh state ```bash diff --git a/stacks/beszel/README.md b/stacks/beszel/README.md index df1e708..c04af76 100644 --- a/stacks/beszel/README.md +++ b/stacks/beszel/README.md @@ -22,17 +22,16 @@ filesystem samples verified; fleet 13/14 up with known fv-ml1 outage. | esh-docker-vm (hub name esh-vm-docker) | beszel-agent-esh | /mnt/backup, /mnt/books | | irv-ml1 | beszel-agent-irv | /worktank, /storetank, /mnt/smithy | | vm-esh-nas | beszel-agent-esh-nas | /mnt/books, /mnt/share, /mnt/music, /mnt/media | -| nh3-dev | beszel | /mnt/backup, /mnt/smithy | +| nh3-dev | beszel | none since 2026-09-25 (was /mnt/backup, /mnt/smithy — see below) | | esh-ml1 (added 2026-09-25) | beszel | none — NVIDIA image (`hosts/esh-ml1.yaml`) for the RTX 2000E Ada | ⚠ **nh3-dev's agent was DOWN from the 2026-09-24 NH3 power recovery until 2026-09-25.** Docker could not bind `/mnt/smithy` at boot ("no such device"): since `1cbde50` the NAS shares are automounted, and nh3-nas was not up yet. Docker does not retry a container that fails to *create*, so `unless-stopped` never brought it back. -Started by hand. **It will recur on the next NH3 cold start** until the agent stops -depending on NAS mounts at boot. The NAS capacity is already reported by -nh3-nas's own agent, so dropping the two extra filesystems here is the simplest fix. -Open item. +Started by hand, then **fixed the same day**: `hosts/nh3-dev.yaml` no longer binds +the NAS shares and `BESZEL_EXTRA_FS` is empty. Their capacity is nh3-nas's own +volume, which nh3-nas's agent reports. Use `infra-ops@` with passwordless sudo, except vm-esh-nas: `lkraven@10.0.50.154` has Docker access. Irvine's hub address is @@ -71,7 +70,7 @@ The reusable playbook is `playbooks/beszel-filesystems.yaml` with `stack_dir`, `host_name`, and `extra_fs` variables. Environment backups are kept in `.env.before-fleet-wiring-20260910` on each host. -nh3-dev has the older `docker-compose` command; use that spelling. It also +nh3-dev now has the `docker compose` plugin (v5.5.1, 2026-09-25); the old `docker-compose` binary is gone. It also requires the external `traefik-net` network to exist even for the agent profile. Only the Beszel agent is started there. On other hosts, use `docker compose`. diff --git a/stacks/beszel/hosts/nh3-dev.yaml b/stacks/beszel/hosts/nh3-dev.yaml index 8bdd1ac..544fc18 100644 --- a/stacks/beszel/hosts/nh3-dev.yaml +++ b/stacks/beszel/hosts/nh3-dev.yaml @@ -1,5 +1,11 @@ +# nh3-dev — deliberately NO extra filesystems (changed 2026-09-25). +# +# This override used to bind /mnt/backup and /mnt/smithy, two nh3-nas NFS +# shares. Since 1cbde50 those are automounted on first access, and on an NH3 cold +# start the NAS is the slowest box up. Docker then failed to bind the share +# ("no such device"), and it never retries a container that fails to CREATE, so +# the agent stayed dead from the 2026-09-24 power recovery until it was noticed. +# Their capacity is nh3-nas's own volume, which nh3-nas's agent already reports, +# so dropping them loses no information and removes the agent's dependency on the NAS. services: - beszel-agent: - volumes: - - /mnt/backup:/extra-filesystems/backup:ro - - /mnt/smithy:/extra-filesystems/smithy:ro + beszel-agent: {}