feat(nh3-pve): prepare for GPU install — pin NIC names by MAC, pull AMT port from vmbr0

nh3-pve and esh-pve are the same Minisforum MS-01 (BIOS AHWSA.1.17). With a card in the x16 slot its root port takes bus 01 and every NIC moves down a bus (measured on esh-pve), so predictable names change (enp2s0f0np0 -> enp3s0f0np0 etc.) and vmbr0 would boot with no uplink. systemd .link files now pin all NICs by MAC, baked into every initramfs and synced to the ESP; udev confirms the files apply. The AMT-capable I226-LM (enp88s0) leaves vmbr0's bridge-ports in the file (next boot), so cabling it for AMT cannot loop the STP-less bridge. Also: documented the NanoKVM (https://10.100.250.171) as nh3-pve's console OOB and that AMT is not wired; nh3-dev's Beszel agent no longer binds NAS shares (it died on the last NH3 cold start); post-boot checklist in persistent-memory.
2026-09-25 10:56:17 -07:00
parent 20039bcda2
commit 2118449881
5 changed files with 179 additions and 10 deletions
@@ -0,0 +1,91 @@
+# nh3-pve — pin NIC names by MAC BEFORE a GPU goes into the PCIe slot, and take
+# the AMT-capable port out of vmbr0.
+#
+# WHY (measured 2026-09-25, not assumed): nh3-pve and esh-pve are the same box —
+# Minisforum MS-01 ("Venus Series"), BIOS AHWSA.1.17. On esh-pve, which has a GPU,
+# the x16 slot's root port 00:01.0 takes PCI bus 01 and every device behind it
+# moves down one bus: X710 at 03:00, I226-V 58, I226-LM 59. On nh3-pve (no GPU)
+# 00:01.0 does not exist: X710 02:00, I226-V 57, I226-LM 58. Predictable NIC names
+# encode the bus, so fitting a GPU renames them:
+#     enp2s0f0np0 -> enp3s0f0np0   (the ONLY connected uplink, SFP+)
+#     enp87s0 -> enp88s0, enp88s0 -> enp89s0, wlp89s0 -> wlp90s0
+# vmbr0's bridge-ports name the old spellings, so the host would boot with NO
+# uplink: the NH3 post office, nh3-dev, DNS and the mesh subnet router all go dark.
+# The fix is a systemd .link file per NIC that matches the MAC (which does not
+# change) and keeps today's name. Nothing changes on the current boot.
+#
+# AMT: the vPro port is the I226-LM (enp88s0, MAC ...:0e). vmbr0 bridges ALL
+# four NICs with STP off, so cabling the LM port while the SFP+ uplink is also
+# connected would put two paths into the same L2 — a switching loop that would
+# take down the site LAN. enp88s0 is removed from bridge-ports, in the FILE only:
+# it takes effect on the next boot (the GPU install), with no live ifreload on the
+# site's hypervisor. enp87s0 stays first in the list so vmbr0 keeps its MAC.
+#
+# Run:  scripts/elway root@nh3-pve --playbook playbooks/nh3-pve-pin-nic-names.yaml
+
+vars:
+  stage_dir: /root/nic-pin-2026-09-25
+
+steps:
+  - name: Back up the network config
+    shell: |
+      mkdir -p {{ stage_dir }}
+      cp -p /etc/network/interfaces {{ stage_dir }}/interfaces.before
+    creates: "{{ stage_dir }}/interfaces.before"
+
+  - name: Pin each NIC's current name to its MAC (systemd .link)
+    shell: |
+      set -e
+      pin() {  # name mac
+        cat > /etc/systemd/network/10-pin-$1.link <<EOF
+      # Pin $1 by MAC so a GPU in the PCIe slot cannot rename it.
+      # See eshpfi playbooks/nh3-pve-pin-nic-names.yaml
+      [Match]
+      MACAddress=$2
+      Type=ether
+      [Link]
+      Name=$1
+      EOF
+      }
+      pin enp2s0f0np0 58:47:ca:76:96:0b
+      pin enp2s0f1np1 58:47:ca:76:96:0c
+      pin enp87s0     58:47:ca:76:96:0d
+      pin enp88s0     58:47:ca:76:96:0e
+      cat > /etc/systemd/network/10-pin-wlp89s0.link <<'EOF'
+      # See eshpfi playbooks/nh3-pve-pin-nic-names.yaml
+      [Match]
+      MACAddress=4c:50:dd:6c:0d:f9
+      [Link]
+      Name=wlp89s0
+      EOF
+    when: "! test -f /etc/systemd/network/10-pin-enp2s0f0np0.link"
+
+  - name: Take the AMT port (enp88s0) out of vmbr0 in the file (next boot)
+    shell: sed -i 's/^\(\s*bridge-ports\) enp87s0 enp88s0 enp2s0f0np0 enp2s0f1np1$/\1 enp87s0 enp2s0f0np0 enp2s0f1np1/' /etc/network/interfaces
+    when: "grep -qE '^\\s*bridge-ports enp87s0 enp88s0 enp2s0f0np0 enp2s0f1np1$' /etc/network/interfaces"
+
+  - name: Rebuild the initramfs so the .link files apply in early boot too
+    # proxmox-boot-tool's post-update hook syncs the ESP.
+    shell: update-initramfs -u -k all
+    when: "! lsinitramfs /boot/initrd.img-$(uname -r) | grep -q '10-pin-enp2s0f0np0.link'"
+
+verify:
+  - name: interfaces file still parses (no live reload)
+    shell: ifreload -a -s
+    changed_when: "false"
+
+  - name: bridge-ports no longer include the AMT port
+    shell: "grep -qE '^\\s*bridge-ports enp87s0 enp2s0f0np0 enp2s0f1np1$' /etc/network/interfaces"
+    changed_when: "false"
+
+  - name: udev would give every wired NIC its pinned name via OUR .link file
+    shell: |
+      for n in enp2s0f0np0 enp2s0f1np1 enp87s0 enp88s0; do
+        out=$(udevadm test-builtin net_setup_link /sys/class/net/$n 2>&1)
+        echo "$out" | grep -q "10-pin-$n.link" || { echo "$n: not matched by 10-pin-$n.link"; exit 1; }
+      done
+    changed_when: "false"
+
+  - name: .link files are inside the running kernel's initramfs
+    shell: lsinitramfs /boot/initrd.img-$(uname -r) | grep -q '10-pin-enp2s0f0np0.link'
+    changed_when: "false"