CPU: 13th Gen Intel Core i9-13900H
RAM: 62.5 GB
Kernel: 6.8.12-43-pve since 2026-09-25 (was 6.8.12-11; see the kernel bullet below). PVE 8.4.1, well behind esh-pve's 8.4.20 (177 packages pending)
Storage: mostly networked — /mnt/pve/pfi-nh3-nas (42 TB) mounted from the Synology at 10.100.50.50:/volume1/VMStorage; ~27 TB used

What it runs

Guests as of 2026-09-24 (qm list / pct list is the live truth). Expected state after a power loss is the column that matters in a recovery: a guest listed as stays off is not a casualty.

ID	guest	kind	onboot	expected after power loss
100	nh3-docker (`10.100.50.40`)	VM	1	up
101	nh3-extdev (`10.100.50.42`)	VM	1	up
102	nh3-dev (`10.100.10.50`)	VM	1	up
104	nh3-laser	VM	—	stays off — on-demand (Prime, 2026-09-24)
105	pbs-nh3 (`10.100.50.90`)	VM	1	up (onboot added 2026-09-24; it had none and stayed down after the outage)
108	opnsense-lab	VM	0	stays off (lab)
103	nh3-wg	CT	1	up
106	nh3-headscale	CT	1	up
107	nh3-scale	CT	1	up (mesh subnet router + fleet egress proxy)
109	nh3-ml1 (`10.100.50.80`)	CT	1 (order 30)	up. GPU LXC, second embed/rerank backend (`servers/nh3-ml1/README.md`); needs the NVIDIA module, so it is the one guest a driver fault can stop

Power-loss recovery (2026-09-24 outage). Every guest boots at once, and nh3-nas is the slowest to serve NFS. NFS clients now mount nh3-nas shares on first access (playbooks/nh3-nfs-automount.yaml, applied to nh3-docker and nh3-dev), so a late NAS no longer leaves shares down. pbs-nh3's datastore line uses bg (background retry) instead. qm guest exec 105 works (the guest agent is on), which is the root path into pbs-nh3; infra-ops is not provisioned there. pfi-gx10 (10.100.50.60) is bare metal on this site, not a guest here, and did not power back on by itself.

Hardware identity, OOB, and the GPU-slot NIC-rename trap (2026-09-25)

Box: Minisforum MS-01 ("Venus Series", BIOS AHWSA.1.17): the same model and BIOS as esh-pve. i9-13900H, X710 2×SFP+, I226-V + I226-LM (vPro/AMT), one PCIe x16 slot (x8 electrical, low-profile, single-slot, slot power only).
Uplink: one SFP+ 10G port has a cable. Since the 2026-09-25 GPU install it is enp2s0f1np1 (MAC …:96:0c, PCI 03:00.1). Before that it was enp2s0f0np0 (…:96:0b), so the cable moved cages during the visit. The pin follows MAC, so this is a physical move and not a rename. Both SFP+ ports are vmbr0 members, so either cage works. Never drop either port from the bridge without checking which one has carrier (ip -br link). The bridge carries the I226-V's MAC …:96:0d because it is the first port listed.
AMT: LIVE at https://10.100.0.151:16993 (nh3-pve-amt.nh3.internal), Intel AMT 16.1.25, since 2026-09-25 2224. It uses a self-signed cert and TLS 1.2 with legacy renegotiation, so OpenSSL 3 clients need Options = UnsafeLegacyRenegotiation; browsers cope. 16992 (plain HTTP) is closed. 664 (TLS redirection: SOL/IDER/KVM) is open. It does not answer ping.
- Port: the I226-LM enp88s0 (MAC 58:47:ca:76:96:0e) is cabled to PFI-UDMSE port 6 (Prime; confirmed from the UDM, which lists the AMT MAC on port 6 at 1 Gb). Port 6 is forward: all, meaning untagged nh3-default plus every tagged VLAN, so AMT (untagged) lands on nh3-default 10.100.0.0/24. ⚠ The first pass wrongly placed it on nh3-sw1, for two reasons. The UDM's port table was read 15 s after link-up, before the UDM refreshed it (it polls, and lags about a minute). And nh3-sw1's STP BPDUs, seen on the wire, are flooded through the UDM's switch, so they do not locate the cable.
- Lease: UDM DHCP reservation nh3-pve-amt → 10.100.0.151.
- ⚠ Why it had no IP: Linux had the port DOWN. When igc downs the I226-LM it powers off the PHY, and AMT's link goes with it (measured: "Link detected: no" while down, a 1 Gb link and a DHCP lease within minutes of ip link set up).
  - It was down because we pulled enp88s0 out of vmbr0 on 2026-09-25 to avoid a loop, which left it unconfigured at boot. Our own change caused it.
  - Fix, persistent: auto enp88s0 / iface enp88s0 inet manual in /etc/network/interfaces (PVE's "Autostart"; it stays up with no address and no bridge). /etc/sysctl.d/90-amt-port.conf turns IPv6 off on it, so the host never SLAACs an address or default route onto nh3-default. Backup: /root/interfaces.bak-20260925-amt.
  - Never down this port or remove the auto line, or AMT goes dark.
- Not yet verified: KVM redirection is enabled with User Opt-in = None (this needs the AMT admin password), and AMT keeps its link with the host powered off (this depends on the MEBx power policy).
- Better home, not done (Prime's call): make UDM port 6 native nh3-mgmt (VLAN 250) and drop the tagged VLANs, so AMT alone sits on the management network. It would need a new reservation and a DNS change. The UDM API key can do it.
- MEBx menu: Ctrl+P at boot.
Console OOB exists: a Sipeed NanoKVM is attached (USB 3346:1009 on the host; web UI https://10.100.250.171, switch port 23, nh3-mgmt). It gives video and keyboard, so BIOS, MEBx and a host that booted without network are all reachable remotely. Power control through it is unverified (the MS-01 has no standard ATX front-panel header). ⚠ It captures the iGPU's HDMI, and RTX-class cards output mini-DP only, so keep the BIOS primary display on the iGPU after a GPU install.
⚠ A GPU in the slot RENAMES every NIC. With a card present the slot's root port 00:01.0 takes PCI bus 01 and everything behind it moves down a bus (measured on esh-pve): X710 02→03, I226-V 57→58, I226-LM 58→59, Wi-Fi 59→5a. The names would become enp3s0f0np0, enp88s0, enp89s0, wlp90s0. vmbr0 would then boot with no uplink and take the whole NH3 site's VMs off the network. Pinned 2026-09-25 by MAC with /etc/systemd/network/10-pin-*.link, also baked into every initramfs and synced to the ESP (playbooks/nh3-pve-pin-nic-names.yaml, verified with udevadm test-builtin net_setup_link). If the host ever boots without network anyway: open the NanoKVM console, run ip -br link, and compare names against /etc/network/interfaces (backup in /root/nic-pin-2026-09-25/).
enp88s0 (the AMT port) is not a vmbr0 bridge port (removed 2026-09-25). STP is off, and UDM port 6 carries the same VLANs as the uplink, so bridging it would loop the site LAN. It is held admin-UP on its own instead (AMT bullet above).
GPU: RTX 2000E Ada at 01:00.0 (10de:28b0), installed 2026-09-25. The NIC pins held: the X710 moved to bus 03 and every NIC kept its name. NVIDIA 580.178.04 (open modules, DKMS) has been on the host since 2026-09-25 at 1527 (playbooks/pve-nvidia-host.yaml), with nvidia-persistenced ordered before pve-guests. It serves CT 109 nh3-ml1.
iGPU restored (2026-09-25 visit). With the card in, the BIOS's Auto primary display had hidden the iGPU. That left the NanoKVM (iGPU HDMI) blind and would have blinded AMT KVM too, since AMT captures only the iGPU. It was set on the visit, and since the 1523 boot 00:02.0 is back with boot_vga=1 and i915 loaded. The NanoKVM should have video again (not checked from here). systemctl reboot --firmware-setup works (OsIndicationsSupported bit 0), so nobody has to catch Del at POST. The AMI Setup variables are not runtime-visible, so there is no OS-side BIOS patch. esh-pve, the same MS-01 and BIOS AHWSA.1.17, is the known-good reference.
Secure Boot: OFF since the 2026-09-25 visit (mokutil --sb-state: disabled), which matches esh-pve. While it was ON (lockdown integrity), the DKMS NVIDIA module was refused and the first install rolled itself back. The playbook's pre-flight refuses if it is ever turned back on without an enrolled DKMS MOK.
⚠ Kernel jumped 6.8.12-11 → 6.8.12-43 at the visit reboot, pulled in by our own playbook. At 1419 the headers step ran apt-get install proxmox-headers-6.8. That upgraded the proxmox-kernel-6.8 meta and installed proxmox-kernel-6.8.12-43-pve-signed. Nobody chose the new kernel; it booted because it was the newest. Side effect: every -43 boot oopses in Bluetooth (btmtk_usb_hci_wmt_sync → NULL deref in hci_power_on, the MS-01's MediaTek BT; taint D). It hit on all three -43 boots and on none of the -11 boots. esh-pve on -42 shows the same oops and has run fine, so it is benign so far: only the BT worker dies. The fix is to blacklist btusb on both hypervisors. That is not done, because it only takes effect at the next boot.
Runs warmer than esh-pve, its twin (Beszel, 2026-09-15 → 09-25).
- nh3-pve's CPU package: median of 2-hour averages 76 °C, peak 2-hour average 88 °C (2026-09-23, CPU ~15%).
- esh-pve: medians 52–55 °C, peak 20-min average 78 °C.
- During the 2026-09-25 2100 vzdump, the 1-minute samples reached 90 °C at 13–21% CPU. They were back to 61 °C once the job finished.
- Workload is a confound: nh3-pve usually carries more load, so this does not prove a cooling fault. Worth checking airflow and dust on the next visit.
- TjMax is 100 °C. The Beszel CPU alert on this host is set at >95 °C for 5 minutes.
lxc-pve 6.0.0-1 → 6.0.0-2 (2026-09-25 1533, that one package only). This is Proxmox fix #7006. Without it, runc 1.5 inside a nesting CT fails every docker run. playbooks/gpu-lxc.yaml now upgrades it first.
OOB plan status. Prime ruled on 2026-09-25 via Miranda to HOLD console-blind until the site visit. Target: the NanoKVM moves to pfi-gx10, and this MS-01 uses its own AMT. The visit did IGFX, turned SB off and cabled plus enabled AMT. AMT came up on the network at 2224 once the host stopped downing its port. Still open: confirm AMT KVM shows the console. Only then move the NanoKVM to the gx10.

Refresh state

scripts/refresh-server-info.sh nh3-pve

Same Proxmox-inspect caveat as pfi-pve: snapshots capture host-level detail only. VM/LXC/storage enumeration needs the Proxmox tooling manually.

Placement rule

Hypervisor. New NH-site workloads land here as VMs.

README.md Unescape Escape

nh3-pve

Network

Hardware

What it runs

Hardware identity, OOB, and the GPU-slot NIC-rename trap (2026-09-25)

Refresh state

Placement rule

README.md