vh
2118449881
nh3-pve and esh-pve are the same Minisforum MS-01 (BIOS AHWSA.1.17). With a card in the x16 slot its root port takes bus 01 and every NIC moves down a bus (measured on esh-pve), so predictable names change (enp2s0f0np0 -> enp3s0f0np0 etc.) and vmbr0 would boot with no uplink. systemd .link files now pin all NICs by MAC, baked into every initramfs and synced to the ESP; udev confirms the files apply. The AMT-capable I226-LM (enp88s0) leaves vmbr0's bridge-ports in the file (next boot), so cabling it for AMT cannot loop the STP-less bridge. Also: documented the NanoKVM (https://10.100.250.171) as nh3-pve's console OOB and that AMT is not wired; nh3-dev's Beszel agent no longer binds NAS shares (it died on the last NH3 cold start); post-boot checklist in persistent-memory.
90 lines
4.6 KiB
Markdown
90 lines
4.6 KiB
Markdown
# nh3-pve
|
||
|
||
Proxmox VE hypervisor for the NH3 site (`nh3-vmhost.phasefinal.com`).
|
||
|
||
## Network
|
||
|
||
- **LAN IP:** 10.100.250.60
|
||
- **FQDN:** `nh3-vmhost.phasefinal.com`
|
||
- **Web UI:** `https://10.100.250.60:8006`
|
||
- **SSH:** `root@nh3-pve` (key auth)
|
||
|
||
## Hardware
|
||
|
||
- **CPU:** 13th Gen Intel Core i9-13900H
|
||
- **RAM:** 62.5 GB
|
||
- **Kernel:** `6.8.12-11-pve` (Proxmox 8.x)
|
||
- **Storage:** mostly networked — `/mnt/pve/pfi-nh3-nas` (42 TB) mounted from the Synology at `10.100.50.50:/volume1/VMStorage`; ~27 TB used
|
||
|
||
## What it runs
|
||
|
||
Guests as of 2026-09-24 (`qm list` / `pct list` is the live truth). **Expected state
|
||
after a power loss** is the column that matters in a recovery: a guest listed as
|
||
`stays off` is not a casualty.
|
||
|
||
| ID | guest | kind | onboot | expected after power loss |
|
||
|---|---|---|---|---|
|
||
| 100 | nh3-docker (`10.100.50.40`) | VM | 1 | up |
|
||
| 101 | nh3-extdev (`10.100.50.42`) | VM | 1 | up |
|
||
| 102 | nh3-dev (`10.100.10.50`) | VM | 1 | up |
|
||
| 104 | nh3-laser | VM | — | **stays off — on-demand** (Prime, 2026-09-24) |
|
||
| 105 | pbs-nh3 (`10.100.50.90`) | VM | 1 | up (onboot added 2026-09-24; it had none and stayed down after the outage) |
|
||
| 108 | opnsense-lab | VM | 0 | stays off (lab) |
|
||
| 103 | nh3-wg | CT | 1 | up |
|
||
| 106 | nh3-headscale | CT | 1 | up |
|
||
| 107 | nh3-scale | CT | 1 | up (mesh subnet router + fleet egress proxy) |
|
||
|
||
**Power-loss recovery (2026-09-24 outage).** Every guest boots at once, and
|
||
nh3-nas is the slowest to serve NFS. NFS clients now mount nh3-nas shares on
|
||
first access (`playbooks/nh3-nfs-automount.yaml`, applied to nh3-docker and
|
||
nh3-dev), so a late NAS no longer leaves shares down. pbs-nh3's datastore line
|
||
uses `bg` (background retry) instead. `qm guest exec 105` works (the guest agent is
|
||
on), which is the root path into pbs-nh3; infra-ops is not provisioned there.
|
||
**pfi-gx10 (`10.100.50.60`) is bare metal on this site, not a guest here**, and did
|
||
not power back on by itself.
|
||
|
||
## Hardware identity, OOB, and the GPU-slot NIC-rename trap (2026-09-25)
|
||
|
||
- **Box:** Minisforum MS-01 ("Venus Series", BIOS `AHWSA.1.17`): the same model
|
||
and BIOS as esh-pve. i9-13900H, X710 2×SFP+, I226-V + **I226-LM (vPro/AMT)**,
|
||
one PCIe x16 slot (x8 electrical, low-profile, single-slot, slot power only).
|
||
- **Uplink:** only `enp2s0f0np0` (SFP+ 10G) has a cable. The bridge carries the
|
||
I226-V's MAC `…:96:0d` because it is the first port listed.
|
||
- **AMT: NOT wired.** The AMT-capable I226-LM (`enp88s0`, MAC `58:47:ca:76:96:0e`)
|
||
has no cable. The ME is present (`/dev/mei0`, "AMT SOL Redirection" 00:16.3), but
|
||
MEBx provisioning status is unknown. To wire it: cable the LM port, then at boot
|
||
press Ctrl+P → set the MEBx password, enable manageability, set network (static
|
||
or DHCP), KVM on, User Opt-in = None, activate network access. MEBx can be driven
|
||
remotely through the NanoKVM below.
|
||
- **Console OOB exists: a Sipeed NanoKVM** is attached (USB `3346:1009` on the host;
|
||
web UI **`https://10.100.250.171`**, switch port 23, nh3-mgmt). It gives video and
|
||
keyboard, so BIOS, MEBx and a host that booted without network are all reachable
|
||
remotely. Power control through it is unverified (the MS-01 has no standard ATX
|
||
front-panel header). ⚠ It captures the **iGPU's HDMI**, and RTX-class cards output
|
||
mini-DP only, so keep the BIOS primary display on the iGPU after a GPU install.
|
||
- ⚠ **A GPU in the slot RENAMES every NIC.** With a card present the slot's root
|
||
port 00:01.0 takes PCI bus 01 and everything behind it moves down a bus (measured
|
||
on esh-pve): X710 02→03, I226-V 57→58, I226-LM 58→59, Wi-Fi 59→5a. The names
|
||
would become `enp3s0f0np0`, `enp88s0`, `enp89s0`, `wlp90s0`. vmbr0 would then boot
|
||
with no uplink and take the whole NH3 site's VMs off the network.
|
||
**Pinned 2026-09-25** by MAC with `/etc/systemd/network/10-pin-*.link`, also baked
|
||
into every initramfs and synced to the ESP (`playbooks/nh3-pve-pin-nic-names.yaml`,
|
||
verified with `udevadm test-builtin net_setup_link`). If the host ever boots
|
||
without network anyway: open the NanoKVM console, run `ip -br link`, and compare
|
||
names against `/etc/network/interfaces` (backup in `/root/nic-pin-2026-09-25/`).
|
||
- **`enp88s0` (the AMT port) is no longer a vmbr0 bridge port** (file edited
|
||
2026-09-25, effective next boot). STP is off, so bridging a second cabled uplink
|
||
into the same L2 would loop the site LAN.
|
||
|
||
## Refresh state
|
||
|
||
```bash
|
||
scripts/refresh-server-info.sh nh3-pve
|
||
```
|
||
|
||
Same Proxmox-inspect caveat as `pfi-pve`: snapshots capture host-level detail only. VM/LXC/storage enumeration needs the Proxmox tooling manually.
|
||
|
||
## Placement rule
|
||
|
||
Hypervisor. New NH-site workloads land here as VMs.
|