vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
59f5474bc718960235b786b07fdf330d93f0d711
esh-pfi-infrastructure/servers/nh3-pve
T
History

README.md

nh3-pve

Proxmox VE hypervisor for the NH3 site (nh3-vmhost.phasefinal.com).

Network

  • LAN IP: 10.100.250.60
  • FQDN: nh3-vmhost.phasefinal.com
  • Web UI: https://10.100.250.60:8006
  • SSH: root@nh3-pve (key auth)

Hardware

  • CPU: 13th Gen Intel Core i9-13900H
  • RAM: 62.5 GB
  • Kernel: 6.8.12-11-pve (Proxmox 8.x)
  • Storage: mostly networked — /mnt/pve/pfi-nh3-nas (42 TB) mounted from the Synology at 10.100.50.50:/volume1/VMStorage; ~27 TB used

What it runs

Guests as of 2026-09-24 (qm list / pct list is the live truth). Expected state after a power loss is the column that matters in a recovery: a guest listed as stays off is not a casualty.

ID guest kind onboot expected after power loss
100 nh3-docker (10.100.50.40) VM 1 up
101 nh3-extdev (10.100.50.42) VM 1 up
102 nh3-dev (10.100.10.50) VM 1 up
104 nh3-laser VM — stays off — on-demand (Prime, 2026-09-24)
105 pbs-nh3 (10.100.50.90) VM 1 up (onboot added 2026-09-24; it had none and stayed down after the outage)
108 opnsense-lab VM 0 stays off (lab)
103 nh3-wg CT 1 up
106 nh3-headscale CT 1 up
107 nh3-scale CT 1 up (mesh subnet router + fleet egress proxy)

Power-loss recovery (2026-09-24 outage). Every guest boots at once, and nh3-nas is the slowest to serve NFS. NFS clients now mount nh3-nas shares on first access (playbooks/nh3-nfs-automount.yaml, applied to nh3-docker and nh3-dev), so a late NAS no longer leaves shares down. pbs-nh3's datastore line uses bg (background retry) instead. qm guest exec 105 works (the guest agent is on), which is the root path into pbs-nh3; infra-ops is not provisioned there. pfi-gx10 (10.100.50.60) is bare metal on this site, not a guest here, and did not power back on by itself.

Hardware identity, OOB, and the GPU-slot NIC-rename trap (2026-09-25)

  • Box: Minisforum MS-01 ("Venus Series", BIOS AHWSA.1.17): the same model and BIOS as esh-pve. i9-13900H, X710 2×SFP+, I226-V + I226-LM (vPro/AMT), one PCIe x16 slot (x8 electrical, low-profile, single-slot, slot power only).
  • Uplink: one SFP+ 10G port has a cable. Since the 2026-09-25 GPU install it is enp2s0f1np1 (MAC …:96:0c, PCI 03:00.1). Before that it was enp2s0f0np0 (…:96:0b), so the cable moved cages during the visit. The pin follows MAC, so this is a physical move and not a rename. Both SFP+ ports are vmbr0 members, so either cage works. Never drop either port from the bridge without checking which one has carrier (ip -br link). The bridge carries the I226-V's MAC …:96:0d because it is the first port listed.
  • AMT: NOT wired. The AMT-capable I226-LM (enp88s0, MAC 58:47:ca:76:96:0e) has no cable. The ME is present (/dev/mei0, "AMT SOL Redirection" 00:16.3), but MEBx provisioning status is unknown. To wire it: cable the LM port, then at boot press Ctrl+P → set the MEBx password, enable manageability, set network (static or DHCP), KVM on, User Opt-in = None, activate network access. MEBx can be driven remotely through the NanoKVM below.
  • Console OOB exists: a Sipeed NanoKVM is attached (USB 3346:1009 on the host; web UI https://10.100.250.171, switch port 23, nh3-mgmt). It gives video and keyboard, so BIOS, MEBx and a host that booted without network are all reachable remotely. Power control through it is unverified (the MS-01 has no standard ATX front-panel header). ⚠ It captures the iGPU's HDMI, and RTX-class cards output mini-DP only, so keep the BIOS primary display on the iGPU after a GPU install.
  • ⚠ A GPU in the slot RENAMES every NIC. With a card present the slot's root port 00:01.0 takes PCI bus 01 and everything behind it moves down a bus (measured on esh-pve): X710 02→03, I226-V 57→58, I226-LM 58→59, Wi-Fi 59→5a. The names would become enp3s0f0np0, enp88s0, enp89s0, wlp90s0. vmbr0 would then boot with no uplink and take the whole NH3 site's VMs off the network. Pinned 2026-09-25 by MAC with /etc/systemd/network/10-pin-*.link, also baked into every initramfs and synced to the ESP (playbooks/nh3-pve-pin-nic-names.yaml, verified with udevadm test-builtin net_setup_link). If the host ever boots without network anyway: open the NanoKVM console, run ip -br link, and compare names against /etc/network/interfaces (backup in /root/nic-pin-2026-09-25/).
  • enp88s0 (the AMT port) is no longer a vmbr0 bridge port (file edited 2026-09-25, effective next boot). STP is off, so bridging a second cabled uplink into the same L2 would loop the site LAN.
  • GPU installed 2026-09-25: RTX 2000E Ada at 01:00.0 (10de:28b0). The pins held: the X710 moved to bus 03 and every NIC kept its name. No NVIDIA driver yet, so nouveau binds it (gsp ctor failed: -2 is expected without GSP firmware). ⚠ After the install the iGPU is gone from the PCI bus: 00:02.0 enumerated on the 2026-09-24 boot and is absent now, /dev/dri does not exist, and the RTX is boot_vga=1. The BIOS's Auto primary display picked the PCIe card and hid the iGPU, so the NanoKVM's iGPU-HDMI capture has no source. Fix: BIOS → Primary Display = IGFX (or enable iGPU Multi-Monitor). Reaching the BIOS now needs a display on the RTX's mini-DP, for example the NanoKVM through a mini-DP→HDMI adapter. Verify with lspci | grep 00:02.0 and boot_vga on 00:02.0.
  • OOB decision (Prime via Miranda, 2026-09-25): HOLD. nh3-pve stays console-blind until the next NH3 site visit. No BIOS change and no reboot until then. The accepted risk: a boot without network means a site trip. Target end state: the NanoKVM moves to pfi-gx10, and this MS-01 gets out-of-band access through its own vPro/AMT (cable the I226-LM enp88s0 and provision MEBx; see the AMT bullet above). ⚠ Do the BIOS fix on that same visit. AMT's KVM redirection captures only the Intel iGPU's framebuffer, so with the iGPU hidden as it is now, AMT KVM would be just as blind as the NanoKVM. Order: set Primary Display = IGFX, provision MEBx, confirm AMT KVM shows the console, and only then move the NanoKVM to the gx10. Reaching the BIOS (2026-09-25 facts). The NanoKVM input is HDMI, the RTX has mini-DP only, and the adapters on hand are mini-DP→DP only. The paths: (1) an active mini-DP→HDMI adapter to feed the NanoKVM, since a passive one depends on DP++; (2) a DisplayPort monitor plus a USB keyboard at the rack; (3) pull the card, boot on the iGPU, set IGFX explicitly (not Auto), then refit the card. The NIC pins make a card-out boot safe. systemctl reboot --firmware-setup works here (OsIndicationsSupported bit 0 set), so nobody has to catch the Del key at POST. An OS-side patch is not possible: the AMI Setup/SaSetup variables are not runtime-visible here, unlike on the gx10. Known-good reference: esh-pve. Same MS-01, same BIOS AHWSA.1.17, same RTX 2000E, and its 00:02.0 is present with boot_vga=1. The target state works on this hardware.

Refresh state

scripts/refresh-server-info.sh nh3-pve

Same Proxmox-inspect caveat as pfi-pve: snapshots capture host-level detail only. VM/LXC/storage enumeration needs the Proxmox tooling manually.

Placement rule

Hypervisor. New NH-site workloads land here as VMs.

Reference in New Issue View Git Blame Copy Permalink