nh3-pve

Proxmox VE hypervisor for the NH3 site (nh3-vmhost.phasefinal.com).

Network

LAN IP: 10.100.250.60
FQDN: nh3-vmhost.phasefinal.com
Web UI: https://10.100.250.60:8006
SSH: root@nh3-pve (key auth)

Hardware

CPU: 13th Gen Intel Core i9-13900H
RAM: 62.5 GB
Kernel: 6.8.12-11-pve (Proxmox 8.x)
Storage: mostly networked — /mnt/pve/pfi-nh3-nas (42 TB) mounted from the Synology at 10.100.50.50:/volume1/VMStorage; ~27 TB used

What it runs

Guests as of 2026-09-24 (qm list / pct list is the live truth). Expected state after a power loss is the column that matters in a recovery: a guest listed as stays off is not a casualty.

ID	guest	kind	onboot	expected after power loss
100	nh3-docker (`10.100.50.40`)	VM	1	up
101	nh3-extdev (`10.100.50.42`)	VM	1	up
102	nh3-dev (`10.100.10.50`)	VM	1	up
104	nh3-laser	VM	—	stays off — on-demand (Prime, 2026-09-24)
105	pbs-nh3 (`10.100.50.90`)	VM	1	up (onboot added 2026-09-24; it had none and stayed down after the outage)
108	opnsense-lab	VM	0	stays off (lab)
103	nh3-wg	CT	1	up
106	nh3-headscale	CT	1	up
107	nh3-scale	CT	1	up (mesh subnet router + fleet egress proxy)

Power-loss recovery (2026-09-24 outage). Every guest boots at once, and nh3-nas is the slowest to serve NFS. NFS clients now mount nh3-nas shares on first access (playbooks/nh3-nfs-automount.yaml, applied to nh3-docker and nh3-dev), so a late NAS no longer leaves shares down. pbs-nh3's datastore line uses bg (background retry) instead. qm guest exec 105 works (the guest agent is on), which is the root path into pbs-nh3; infra-ops is not provisioned there. pfi-gx10 (10.100.50.60) is bare metal on this site, not a guest here, and did not power back on by itself.

Hardware identity, OOB, and the GPU-slot NIC-rename trap (2026-09-25)

Box: Minisforum MS-01 ("Venus Series", BIOS AHWSA.1.17): the same model and BIOS as esh-pve. i9-13900H, X710 2×SFP+, I226-V + I226-LM (vPro/AMT), one PCIe x16 slot (x8 electrical, low-profile, single-slot, slot power only).
Uplink: one SFP+ 10G port has a cable. Since the 2026-09-25 GPU install it is enp2s0f1np1 (MAC …:96:0c, PCI 03:00.1). Before that it was enp2s0f0np0 (…:96:0b), so the cable moved cages during the visit. The pin follows MAC, so this is a physical move and not a rename. Both SFP+ ports are vmbr0 members, so either cage works. Never drop either port from the bridge without checking which one has carrier (ip -br link). The bridge carries the I226-V's MAC …:96:0d because it is the first port listed.
AMT: NOT wired. The AMT-capable I226-LM (enp88s0, MAC 58:47:ca:76:96:0e) has no cable. The ME is present (/dev/mei0, "AMT SOL Redirection" 00:16.3), but MEBx provisioning status is unknown. To wire it: cable the LM port, then at boot press Ctrl+P → set the MEBx password, enable manageability, set network (static or DHCP), KVM on, User Opt-in = None, activate network access. MEBx can be driven remotely through the NanoKVM below.
Console OOB exists: a Sipeed NanoKVM is attached (USB 3346:1009 on the host; web UI https://10.100.250.171, switch port 23, nh3-mgmt). It gives video and keyboard, so BIOS, MEBx and a host that booted without network are all reachable remotely. Power control through it is unverified (the MS-01 has no standard ATX front-panel header). ⚠ It captures the iGPU's HDMI, and RTX-class cards output mini-DP only, so keep the BIOS primary display on the iGPU after a GPU install.
⚠ A GPU in the slot RENAMES every NIC. With a card present the slot's root port 00:01.0 takes PCI bus 01 and everything behind it moves down a bus (measured on esh-pve): X710 02→03, I226-V 57→58, I226-LM 58→59, Wi-Fi 59→5a. The names would become enp3s0f0np0, enp88s0, enp89s0, wlp90s0. vmbr0 would then boot with no uplink and take the whole NH3 site's VMs off the network. Pinned 2026-09-25 by MAC with /etc/systemd/network/10-pin-*.link, also baked into every initramfs and synced to the ESP (playbooks/nh3-pve-pin-nic-names.yaml, verified with udevadm test-builtin net_setup_link). If the host ever boots without network anyway: open the NanoKVM console, run ip -br link, and compare names against /etc/network/interfaces (backup in /root/nic-pin-2026-09-25/).
enp88s0 (the AMT port) is no longer a vmbr0 bridge port (file edited 2026-09-25, effective next boot). STP is off, so bridging a second cabled uplink into the same L2 would loop the site LAN.
GPU installed 2026-09-25: RTX 2000E Ada at 01:00.0 (10de:28b0). The pins held: the X710 moved to bus 03 and every NIC kept its name. No NVIDIA driver yet, so nouveau binds it (gsp ctor failed: -2 is expected without GSP firmware). ⚠ After the install the iGPU is gone from the PCI bus: 00:02.0 enumerated on the 2026-09-24 boot and is absent now, /dev/dri does not exist, and the RTX is boot_vga=1. The BIOS's Auto primary display picked the PCIe card and hid the iGPU, so the NanoKVM's iGPU-HDMI capture has no source. Fix: BIOS → Primary Display = IGFX (or enable iGPU Multi-Monitor). Reaching the BIOS now needs a display on the RTX's mini-DP, for example the NanoKVM through a mini-DP→HDMI adapter. Verify with lspci | grep 00:02.0 and boot_vga on 00:02.0.

Refresh state

scripts/refresh-server-info.sh nh3-pve

Same Proxmox-inspect caveat as pfi-pve: snapshots capture host-level detail only. VM/LXC/storage enumeration needs the Proxmox tooling manually.

Placement rule

Hypervisor. New NH-site workloads land here as VMs.

5.7 KiB Raw Blame History Unescape Escape