feat(nh3-pve): prepare for GPU install — pin NIC names by MAC, pull AMT port from vmbr0

nh3-pve and esh-pve are the same Minisforum MS-01 (BIOS AHWSA.1.17). With a
card in the x16 slot its root port takes bus 01 and every NIC moves down a
bus (measured on esh-pve), so predictable names change (enp2s0f0np0 ->
enp3s0f0np0 etc.) and vmbr0 would boot with no uplink. systemd .link files
now pin all NICs by MAC, baked into every initramfs and synced to the ESP;
udev confirms the files apply. The AMT-capable I226-LM (enp88s0) leaves
vmbr0's bridge-ports in the file (next boot), so cabling it for AMT cannot
loop the STP-less bridge.

Also: documented the NanoKVM (https://10.100.250.171) as nh3-pve's console
OOB and that AMT is not wired; nh3-dev's Beszel agent no longer binds NAS
shares (it died on the last NH3 cold start); post-boot checklist in
persistent-memory.

persistent-memory.md

@@ -117,6 +117,46 @@ no longer deployed sidecars here. See Recent decisions.)
 
 _As of 2026-09-24 ~2245 PT._
 
+### ⭐⭐ IN PROGRESS 2026-09-25 ~1100 PT: nh3-pve shut down for a GPU install (Prime on site)
+
+This session runs ON nh3-dev and dies with it. **Successor: work this list
+first.** (No /tmp handoff: nh3-dev wipes /tmp at boot.)
+
+Pre-flight done before shutdown:
+- NIC names pinned by MAC (a GPU renames every NIC on this MS-01), initramfs
+  rebuilt and ESP synced.
+- AMT port `enp88s0` removed from vmbr0 in the file.
+- nh3-dev's Beszel agent no longer binds NAS shares.
+- Details in `servers/nh3-pve/README.md`.
+
+**Post-boot checklist:**
+1. **Host on the network:** `ping 10.100.250.60`, then `ip -br link` — names
+   unchanged, `enp2s0f0np0` UP inside vmbr0. If there's no network, open the
+   NanoKVM console at **https://10.100.250.171**.
+2. **GPU seen:** `lspci | grep -i nvidia` → expect `01:00.0`, and the X710 moved to
+   `03:00` while keeping its pinned name. The BIOS primary display must stay on the
+   iGPU, or the NanoKVM goes black.
+3. **Guests:**
+   - running: 100 nh3-docker1, 101 nh3-extdev, 102 nh3-dev, 105 pbs-nh3, CT 103
+     nh3-wg, CT 106 nh3-headscale, CT 107 nh3-scale;
+   - staying off (expected): 104 nh3-laser, 108 opnsense-lab.
+4. **NFS:** touch the automounts — nh3-docker `/mnt/compose` `/mnt/backup`, nh3-dev
+   `/mnt/backup` `/mnt/smithy`.
+5. **Post office** reachable (`postbox status`, `mode: push`). Kuma "althing post
+   office" green.
+6. **nh3-dev services:** svos :8770 + hermes-gateway (Miranda's channel), Booth
+   :8090, ttyd/zellij seats, `dev-backup.timer`. Beszel shows nh3-dev /
+   nh3-docker / pbs-nh3 up.
+7. **Claude sessions:** relaunching them is Prime's call.
+8. **pfi-gx10:** if Prime ran the AC-pull test on this visit, verify it came back
+   by itself.
+9. **Next work:** the GPU's purpose is TBD with Prime; likely the second embed/rerank
+   backend. Reuse `playbooks/esh-pve-nvidia-host.yaml` / `esh-ml1-lxc.yaml` (make
+   them host-generic). nh3-pve runs kernel 6.8.12-11 with **no matching
+   proxmox-headers installed**, on PVE 8.4.1 (Debian 12 template only). If the LM
+   port was cabled, MEBx provisioning can be done through the NanoKVM.
+
+
 ### ⭐ esh-ml1 = the fleet's utility GPU host (2026-09-25): embed/rerank (TEI) + reward (vLLM)
 
 TEI is the fleet embed/rerank engine (Prime) and esh-ml1 is the SOLE backend

playbooks/nh3-pve-pin-nic-names.yaml

@@ -0,0 +1,91 @@
+# nh3-pve — pin NIC names by MAC BEFORE a GPU goes into the PCIe slot, and take
+# the AMT-capable port out of vmbr0.
+#
+# WHY (measured 2026-09-25, not assumed): nh3-pve and esh-pve are the same box —
+# Minisforum MS-01 ("Venus Series"), BIOS AHWSA.1.17. On esh-pve, which has a GPU,
+# the x16 slot's root port 00:01.0 takes PCI bus 01 and every device behind it
+# moves down one bus: X710 at 03:00, I226-V 58, I226-LM 59. On nh3-pve (no GPU)
+# 00:01.0 does not exist: X710 02:00, I226-V 57, I226-LM 58. Predictable NIC names
+# encode the bus, so fitting a GPU renames them:
+#     enp2s0f0np0 -> enp3s0f0np0   (the ONLY connected uplink, SFP+)
+#     enp87s0 -> enp88s0, enp88s0 -> enp89s0, wlp89s0 -> wlp90s0
+# vmbr0's bridge-ports name the old spellings, so the host would boot with NO
+# uplink: the NH3 post office, nh3-dev, DNS and the mesh subnet router all go dark.
+# The fix is a systemd .link file per NIC that matches the MAC (which does not
+# change) and keeps today's name. Nothing changes on the current boot.
+#
+# AMT: the vPro port is the I226-LM (enp88s0, MAC ...:0e). vmbr0 bridges ALL
+# four NICs with STP off, so cabling the LM port while the SFP+ uplink is also
+# connected would put two paths into the same L2 — a switching loop that would
+# take down the site LAN. enp88s0 is removed from bridge-ports, in the FILE only:
+# it takes effect on the next boot (the GPU install), with no live ifreload on the
+# site's hypervisor. enp87s0 stays first in the list so vmbr0 keeps its MAC.
+#
+# Run:  scripts/elway root@nh3-pve --playbook playbooks/nh3-pve-pin-nic-names.yaml
+
+vars:
+  stage_dir: /root/nic-pin-2026-09-25
+
+steps:
+  - name: Back up the network config
+    shell: |
+      mkdir -p {{ stage_dir }}
+      cp -p /etc/network/interfaces {{ stage_dir }}/interfaces.before
+    creates: "{{ stage_dir }}/interfaces.before"
+
+  - name: Pin each NIC's current name to its MAC (systemd .link)
+    shell: |
+      set -e
+      pin() {  # name mac
+        cat > /etc/systemd/network/10-pin-$1.link <<EOF
+      # Pin $1 by MAC so a GPU in the PCIe slot cannot rename it.
+      # See eshpfi playbooks/nh3-pve-pin-nic-names.yaml
+      [Match]
+      MACAddress=$2
+      Type=ether
+      [Link]
+      Name=$1
+      EOF
+      }
+      pin enp2s0f0np0 58:47:ca:76:96:0b
+      pin enp2s0f1np1 58:47:ca:76:96:0c
+      pin enp87s0     58:47:ca:76:96:0d
+      pin enp88s0     58:47:ca:76:96:0e
+      cat > /etc/systemd/network/10-pin-wlp89s0.link <<'EOF'
+      # See eshpfi playbooks/nh3-pve-pin-nic-names.yaml
+      [Match]
+      MACAddress=4c:50:dd:6c:0d:f9
+      [Link]
+      Name=wlp89s0
+      EOF
+    when: "! test -f /etc/systemd/network/10-pin-enp2s0f0np0.link"
+
+  - name: Take the AMT port (enp88s0) out of vmbr0 in the file (next boot)
+    shell: sed -i 's/^\(\s*bridge-ports\) enp87s0 enp88s0 enp2s0f0np0 enp2s0f1np1$/\1 enp87s0 enp2s0f0np0 enp2s0f1np1/' /etc/network/interfaces
+    when: "grep -qE '^\\s*bridge-ports enp87s0 enp88s0 enp2s0f0np0 enp2s0f1np1$' /etc/network/interfaces"
+
+  - name: Rebuild the initramfs so the .link files apply in early boot too
+    # proxmox-boot-tool's post-update hook syncs the ESP.
+    shell: update-initramfs -u -k all
+    when: "! lsinitramfs /boot/initrd.img-$(uname -r) | grep -q '10-pin-enp2s0f0np0.link'"
+
+verify:
+  - name: interfaces file still parses (no live reload)
+    shell: ifreload -a -s
+    changed_when: "false"
+
+  - name: bridge-ports no longer include the AMT port
+    shell: "grep -qE '^\\s*bridge-ports enp87s0 enp2s0f0np0 enp2s0f1np1$' /etc/network/interfaces"
+    changed_when: "false"
+
+  - name: udev would give every wired NIC its pinned name via OUR .link file
+    shell: |
+      for n in enp2s0f0np0 enp2s0f1np1 enp87s0 enp88s0; do
+        out=$(udevadm test-builtin net_setup_link /sys/class/net/$n 2>&1)
+        echo "$out" | grep -q "10-pin-$n.link" || { echo "$n: not matched by 10-pin-$n.link"; exit 1; }
+      done
+    changed_when: "false"
+
+  - name: .link files are inside the running kernel's initramfs
+    shell: lsinitramfs /boot/initrd.img-$(uname -r) | grep -q '10-pin-enp2s0f0np0.link'
+    changed_when: "false"

servers/nh3-pve/README.md

@@ -43,6 +43,39 @@ on), which is the root path into pbs-nh3; infra-ops is not provisioned there.
 **pfi-gx10 (`10.100.50.60`) is bare metal on this site, not a guest here**, and did
 not power back on by itself.
 
+## Hardware identity, OOB, and the GPU-slot NIC-rename trap (2026-09-25)
+
+- **Box:** Minisforum MS-01 ("Venus Series", BIOS `AHWSA.1.17`): the same model
+  and BIOS as esh-pve. i9-13900H, X710 2×SFP+, I226-V + **I226-LM (vPro/AMT)**,
+  one PCIe x16 slot (x8 electrical, low-profile, single-slot, slot power only).
+- **Uplink:** only `enp2s0f0np0` (SFP+ 10G) has a cable. The bridge carries the
+  I226-V's MAC `…:96:0d` because it is the first port listed.
+- **AMT: NOT wired.** The AMT-capable I226-LM (`enp88s0`, MAC `58:47:ca:76:96:0e`)
+  has no cable. The ME is present (`/dev/mei0`, "AMT SOL Redirection" 00:16.3), but
+  MEBx provisioning status is unknown. To wire it: cable the LM port, then at boot
+  press Ctrl+P → set the MEBx password, enable manageability, set network (static
+  or DHCP), KVM on, User Opt-in = None, activate network access. MEBx can be driven
+  remotely through the NanoKVM below.
+- **Console OOB exists: a Sipeed NanoKVM** is attached (USB `3346:1009` on the host;
+  web UI **`https://10.100.250.171`**, switch port 23, nh3-mgmt). It gives video and
+  keyboard, so BIOS, MEBx and a host that booted without network are all reachable
+  remotely. Power control through it is unverified (the MS-01 has no standard ATX
+  front-panel header). ⚠ It captures the **iGPU's HDMI**, and RTX-class cards output
+  mini-DP only, so keep the BIOS primary display on the iGPU after a GPU install.
+- ⚠ **A GPU in the slot RENAMES every NIC.** With a card present the slot's root
+  port 00:01.0 takes PCI bus 01 and everything behind it moves down a bus (measured
+  on esh-pve): X710 02→03, I226-V 57→58, I226-LM 58→59, Wi-Fi 59→5a. The names
+  would become `enp3s0f0np0`, `enp88s0`, `enp89s0`, `wlp90s0`. vmbr0 would then boot
+  with no uplink and take the whole NH3 site's VMs off the network.
+  **Pinned 2026-09-25** by MAC with `/etc/systemd/network/10-pin-*.link`, also baked
+  into every initramfs and synced to the ESP (`playbooks/nh3-pve-pin-nic-names.yaml`,
+  verified with `udevadm test-builtin net_setup_link`). If the host ever boots
+  without network anyway: open the NanoKVM console, run `ip -br link`, and compare
+  names against `/etc/network/interfaces` (backup in `/root/nic-pin-2026-09-25/`).
+- **`enp88s0` (the AMT port) is no longer a vmbr0 bridge port** (file edited
+  2026-09-25, effective next boot). STP is off, so bridging a second cabled uplink
+  into the same L2 would loop the site LAN.
+
 ## Refresh state
 
 ```bash

stacks/beszel/README.md

@@ -22,17 +22,16 @@ filesystem samples verified; fleet 13/14 up with known fv-ml1 outage.
 | esh-docker-vm (hub name esh-vm-docker) | beszel-agent-esh | /mnt/backup, /mnt/books |
 | irv-ml1 | beszel-agent-irv | /worktank, /storetank, /mnt/smithy |
 | vm-esh-nas | beszel-agent-esh-nas | /mnt/books, /mnt/share, /mnt/music, /mnt/media |
-| nh3-dev | beszel | /mnt/backup, /mnt/smithy |
+| nh3-dev | beszel | none since 2026-09-25 (was /mnt/backup, /mnt/smithy — see below) |
 | esh-ml1 (added 2026-09-25) | beszel | none — NVIDIA image (`hosts/esh-ml1.yaml`) for the RTX 2000E Ada |
 
 ⚠ **nh3-dev's agent was DOWN from the 2026-09-24 NH3 power recovery until
 2026-09-25.** Docker could not bind `/mnt/smithy` at boot ("no such device"): since
 `1cbde50` the NAS shares are automounted, and nh3-nas was not up yet. Docker does not
 retry a container that fails to *create*, so `unless-stopped` never brought it back.
-Started by hand. **It will recur on the next NH3 cold start** until the agent stops
-depending on NAS mounts at boot. The NAS capacity is already reported by
-nh3-nas's own agent, so dropping the two extra filesystems here is the simplest fix.
-Open item.
+Started by hand, then **fixed the same day**: `hosts/nh3-dev.yaml` no longer binds
+the NAS shares and `BESZEL_EXTRA_FS` is empty. Their capacity is nh3-nas's own
+volume, which nh3-nas's agent reports.
 
 Use `infra-ops@<ip>` with passwordless sudo, except vm-esh-nas:
 `lkraven@10.0.50.154` has Docker access. Irvine's hub address is
@@ -71,7 +70,7 @@ The reusable playbook is `playbooks/beszel-filesystems.yaml` with `stack_dir`,
 `host_name`, and `extra_fs` variables. Environment backups are kept in
 `.env.before-fleet-wiring-20260910` on each host.
 
-nh3-dev has the older `docker-compose` command; use that spelling. It also
+nh3-dev now has the `docker compose` plugin (v5.5.1, 2026-09-25); the old `docker-compose` binary is gone. It also
 requires the external `traefik-net` network to exist even for the agent profile.
 Only the Beszel agent is started there. On other hosts, use `docker compose`.
 

stacks/beszel/hosts/nh3-dev.yaml

@@ -1,5 +1,11 @@
+# nh3-dev — deliberately NO extra filesystems (changed 2026-09-25).
+#
+# This override used to bind /mnt/backup and /mnt/smithy, two nh3-nas NFS
+# shares. Since 1cbde50 those are automounted on first access, and on an NH3 cold
+# start the NAS is the slowest box up. Docker then failed to bind the share
+# ("no such device"), and it never retries a container that fails to CREATE, so
+# the agent stayed dead from the 2026-09-24 power recovery until it was noticed.
+# Their capacity is nh3-nas's own volume, which nh3-nas's agent already reports,
+# so dropping them loses no information and removes the agent's dependency on the NAS.
 services:
-  beszel-agent:
-    volumes:
-      - /mnt/backup:/extra-filesystems/backup:ro
-      - /mnt/smithy:/extra-filesystems/smithy:ro
+  beszel-agent: {}