esh-pfi-infrastructure/servers/nh3-pve/README.md

# nh3-pve

Proxmox VE hypervisor for the NH3 site (`nh3-vmhost.phasefinal.com`).

## Network

- **LAN IP:** 10.100.250.60
- **FQDN:** `nh3-vmhost.phasefinal.com`
- **Web UI:** `https://10.100.250.60:8006`
- **SSH:** `root@nh3-pve` (key auth)

## Hardware

- **CPU:** 13th Gen Intel Core i9-13900H
- **RAM:** 62.5 GB
- **Kernel:** `6.8.12-11-pve` (Proxmox 8.x)
- **Storage:** mostly networked — `/mnt/pve/pfi-nh3-nas` (42 TB) mounted from the Synology at `10.100.50.50:/volume1/VMStorage`; ~27 TB used

## What it runs

Guests as of 2026-09-24 (`qm list` / `pct list` is the live truth). **Expected state
after a power loss** is the column that matters in a recovery: a guest listed as
`stays off` is not a casualty.

| ID | guest | kind | onboot | expected after power loss |
|---|---|---|---|---|
| 100 | nh3-docker (`10.100.50.40`) | VM | 1 | up |
| 101 | nh3-extdev (`10.100.50.42`) | VM | 1 | up |
| 102 | nh3-dev (`10.100.10.50`) | VM | 1 | up |
| 104 | nh3-laser | VM | — | **stays off — on-demand** (Prime, 2026-09-24) |
| 105 | pbs-nh3 (`10.100.50.90`) | VM | 1 | up (onboot added 2026-09-24; it had none and stayed down after the outage) |
| 108 | opnsense-lab | VM | 0 | stays off (lab) |
| 103 | nh3-wg | CT | 1 | up |
| 106 | nh3-headscale | CT | 1 | up |
| 107 | nh3-scale | CT | 1 | up (mesh subnet router + fleet egress proxy) |

**Power-loss recovery (2026-09-24 outage).** Every guest boots at once, and
nh3-nas is the slowest to serve NFS. NFS clients now mount nh3-nas shares on
first access (`playbooks/nh3-nfs-automount.yaml`, applied to nh3-docker and
nh3-dev), so a late NAS no longer leaves shares down. pbs-nh3's datastore line
uses `bg` (background retry) instead. `qm guest exec 105` works (the guest agent is
on), which is the root path into pbs-nh3; infra-ops is not provisioned there.
**pfi-gx10 (`10.100.50.60`) is bare metal on this site, not a guest here**, and did
not power back on by itself.

## Hardware identity, OOB, and the GPU-slot NIC-rename trap (2026-09-25)

- **Box:** Minisforum MS-01 ("Venus Series", BIOS `AHWSA.1.17`): the same model
  and BIOS as esh-pve. i9-13900H, X710 2×SFP+, I226-V + **I226-LM (vPro/AMT)**,
  one PCIe x16 slot (x8 electrical, low-profile, single-slot, slot power only).
- **Uplink:** one SFP+ 10G port has a cable. Since the 2026-09-25 GPU install it
  is **`enp2s0f1np1`** (MAC `…:96:0c`, PCI 03:00.1). Before that it was
  `enp2s0f0np0` (`…:96:0b`), so the cable moved cages during the visit. The pin
  follows MAC, so this is a physical move and not a rename. Both SFP+ ports are
  vmbr0 members, so either cage works. **Never drop either port from the bridge**
  without checking which one has carrier (`ip -br link`). The bridge carries the
  I226-V's MAC `…:96:0d` because it is the first port listed.
- **AMT: NOT wired.** The AMT-capable I226-LM (`enp88s0`, MAC `58:47:ca:76:96:0e`)
  has no cable. The ME is present (`/dev/mei0`, "AMT SOL Redirection" 00:16.3), but
  MEBx provisioning status is unknown. To wire it: cable the LM port, then at boot
  press Ctrl+P → set the MEBx password, enable manageability, set network (static
  or DHCP), KVM on, User Opt-in = None, activate network access. MEBx can be driven
  remotely through the NanoKVM below.
- **Console OOB exists: a Sipeed NanoKVM** is attached (USB `3346:1009` on the host;
  web UI **`https://10.100.250.171`**, switch port 23, nh3-mgmt). It gives video and
  keyboard, so BIOS, MEBx and a host that booted without network are all reachable
  remotely. Power control through it is unverified (the MS-01 has no standard ATX
  front-panel header). ⚠ It captures the **iGPU's HDMI**, and RTX-class cards output
  mini-DP only, so keep the BIOS primary display on the iGPU after a GPU install.
- ⚠ **A GPU in the slot RENAMES every NIC.** With a card present the slot's root
  port 00:01.0 takes PCI bus 01 and everything behind it moves down a bus (measured
  on esh-pve): X710 02→03, I226-V 57→58, I226-LM 58→59, Wi-Fi 59→5a. The names
  would become `enp3s0f0np0`, `enp88s0`, `enp89s0`, `wlp90s0`. vmbr0 would then boot
  with no uplink and take the whole NH3 site's VMs off the network.
  **Pinned 2026-09-25** by MAC with `/etc/systemd/network/10-pin-*.link`, also baked
  into every initramfs and synced to the ESP (`playbooks/nh3-pve-pin-nic-names.yaml`,
  verified with `udevadm test-builtin net_setup_link`). If the host ever boots
  without network anyway: open the NanoKVM console, run `ip -br link`, and compare
  names against `/etc/network/interfaces` (backup in `/root/nic-pin-2026-09-25/`).
- **`enp88s0` (the AMT port) is no longer a vmbr0 bridge port** (file edited
  2026-09-25, effective next boot). STP is off, so bridging a second cabled uplink
  into the same L2 would loop the site LAN.
- **GPU installed 2026-09-25: RTX 2000E Ada at `01:00.0`** (`10de:28b0`). The pins
  held: the X710 moved to bus 03 and every NIC kept its name. No NVIDIA driver yet,
  so `nouveau` binds it (`gsp ctor failed: -2` is expected without GSP firmware).
  ⚠ **After the install the iGPU is gone from the PCI bus**: `00:02.0` enumerated on
  the 2026-09-24 boot and is absent now, `/dev/dri` does not exist, and the RTX is
  `boot_vga=1`. The BIOS's Auto primary display picked the PCIe card and hid the
  iGPU, so the NanoKVM's iGPU-HDMI capture has no source. Fix: BIOS → Primary
  Display = IGFX (or enable iGPU Multi-Monitor). Reaching the BIOS now needs a
  display on the RTX's mini-DP, for example the NanoKVM through a mini-DP→HDMI
  adapter. Verify with `lspci | grep 00:02.0` and `boot_vga` on `00:02.0`.

## Refresh state

```bash
scripts/refresh-server-info.sh nh3-pve
```

Same Proxmox-inspect caveat as `pfi-pve`: snapshots capture host-level detail only. VM/LXC/storage enumeration needs the Proxmox tooling manually.

## Placement rule

Hypervisor. New NH-site workloads land here as VMs.