esh-pfi-infrastructure/servers/nh3-pve/README.md

# nh3-pve

Proxmox VE hypervisor for the NH3 site (`nh3-vmhost.phasefinal.com`).

## Network

- **LAN IP:** 10.100.250.60
- **FQDN:** `nh3-vmhost.phasefinal.com`
- **Web UI:** `https://10.100.250.60:8006`
- **SSH:** `root@nh3-pve` (key auth)

## Hardware

- **CPU:** 13th Gen Intel Core i9-13900H
- **RAM:** 62.5 GB
- **Kernel:** `6.8.12-43-pve` since 2026-09-25 (was `6.8.12-11`; see the kernel bullet below). PVE `8.4.1`, well behind esh-pve's `8.4.20` (177 packages pending)
- **Storage:** mostly networked — `/mnt/pve/pfi-nh3-nas` (42 TB) mounted from the Synology at `10.100.50.50:/volume1/VMStorage`; ~27 TB used

## What it runs

Guests as of 2026-09-24 (`qm list` / `pct list` is the live truth). **Expected state
after a power loss** is the column that matters in a recovery: a guest listed as
`stays off` is not a casualty.

| ID | guest | kind | onboot | expected after power loss |
|---|---|---|---|---|
| 100 | nh3-docker (`10.100.50.40`) | VM | 1 | up |
| 101 | nh3-extdev (`10.100.50.42`) | VM | 1 | up |
| 102 | nh3-dev (`10.100.10.50`) | VM | 1 | up |
| 104 | nh3-laser | VM | — | **stays off — on-demand** (Prime, 2026-09-24) |
| 105 | pbs-nh3 (`10.100.50.90`) | VM | 1 | up (onboot added 2026-09-24; it had none and stayed down after the outage) |
| 108 | opnsense-lab | VM | 0 | stays off (lab) |
| 103 | nh3-wg | CT | 1 | up |
| 106 | nh3-headscale | CT | 1 | up |
| 107 | nh3-scale | CT | 1 | up (mesh subnet router + fleet egress proxy) |
| 109 | nh3-ml1 (`10.100.50.80`) | CT | 1 (order 30) | up. GPU LXC, second embed/rerank backend (`servers/nh3-ml1/README.md`); needs the NVIDIA module, so it is the one guest a driver fault can stop |

**Power-loss recovery (2026-09-24 outage).** Every guest boots at once, and
nh3-nas is the slowest to serve NFS. NFS clients now mount nh3-nas shares on
first access (`playbooks/nh3-nfs-automount.yaml`, applied to nh3-docker and
nh3-dev), so a late NAS no longer leaves shares down. pbs-nh3's datastore line
uses `bg` (background retry) instead. `qm guest exec 105` works (the guest agent is
on), which is the root path into pbs-nh3; infra-ops is not provisioned there.
**pfi-gx10 (`10.100.50.60`) is bare metal on this site, not a guest here**, and did
not power back on by itself.

## Hardware identity, OOB, and the GPU-slot NIC-rename trap (2026-09-25)

- **Box:** Minisforum MS-01 ("Venus Series", BIOS `AHWSA.1.17`): the same model
  and BIOS as esh-pve. i9-13900H, X710 2×SFP+, I226-V + **I226-LM (vPro/AMT)**,
  one PCIe x16 slot (x8 electrical, low-profile, single-slot, slot power only).
- **Uplink:** one SFP+ 10G port has a cable. Since the 2026-09-25 GPU install it
  is **`enp2s0f1np1`** (MAC `…:96:0c`, PCI 03:00.1). Before that it was
  `enp2s0f0np0` (`…:96:0b`), so the cable moved cages during the visit. The pin
  follows MAC, so this is a physical move and not a rename. Both SFP+ ports are
  vmbr0 members, so either cage works. **Never drop either port from the bridge**
  without checking which one has carrier (`ip -br link`). The bridge carries the
  I226-V's MAC `…:96:0d` because it is the first port listed.
- **AMT: `https://10.100.250.61:16993`** (`nh3-pve-amt.nh3.internal`; Homepage
  card *NH3-PVE-AMT* under Infra - NH3). Intel AMT 16.1.25, **Admin Control
  Mode**. User `admin`; the password is in the vault as `nh3-pve/amt-admin`.
  - **Static IP since 2026-09-25 2306:** `10.100.250.61/24`, gateway and DNS
    `10.100.250.1`. It is set in AMT through WS-Man, and the UDM reservation
    stays as a placeholder.
    - Why static: AMT does not re-DHCP after a VLAN move. It kept its old
      nh3-default lease, and a link drop makes it send only one ARP for its old
      gateway. Static also means OOB does not depend on DHCP.
    - How it was changed while AMT sat on the old address: a temporary
      `10.100.0.250/32` plus a `/32` route on nh3-pve's vmbr0 (same L2), and an
      ssh tunnel. The temporary address was removed afterwards.
  - **Remote-screen settings, verified by reading them back:**
    - KVM enabled (`CIM_KVMRedirectionSAP` EnabledState 6, ready).
    - Redirection listener on (`AMT_RedirectionService` 32771, ListenerEnabled).
    - **User Opt-in = None** (`IPS_OptInService.OptInRequired` 0), so no 6-digit
      code is needed at the rack.
    - The VNC port 5900 is off. Use MeshCommander.
    - Tool: `scripts/amt-wsman.py`.
  - ⚠ **Before the NanoKVM leaves this box, fit a dummy HDMI plug** on the iGPU
    HDMI. AMT KVM draws only an active iGPU output. MS-01 owners commonly report
    a black KVM screen without one.
  - It uses a self-signed cert and TLS 1.2 with legacy renegotiation, so OpenSSL 3
    clients need `Options = UnsafeLegacyRenegotiation`; browsers cope. 16992
    (plain HTTP) is closed. 664 (TLS redirection: SOL/IDER/KVM) is open. It does
    not answer ping.
  - **Port:** the I226-LM `enp88s0` (MAC `58:47:ca:76:96:0e`) is cabled to
    **PFI-UDMSE port 6** (Prime; confirmed from the UDM, which lists the AMT MAC
    on port 6 at 1 Gb). Since 2237 on 2026-09-25 the port override is
    **native nh3-mgmt, tagged VLANs blocked** (name "nh3-pve AMT"). It was
    `forward: all`, with untagged nh3-default.
    Reservation `nh3-pve-amt` → `10.100.250.61`.
  - **ARP guard:** port 6 now puts `enp88s0` on the same untagged L2 as vmbr0
    (`10.100.250.60`). Without a guard, Linux would answer ARP for `.60` out of
    `enp88s0` with MAC `…:0e`, which AMT shares, and the site would see nh3-pve at
    two MACs. `/etc/sysctl.d/90-amt-port.conf` sets `arp_ignore=8` and
    `arp_announce=2` on `enp88s0`, next to the IPv6 disable.
    ⚠ The first pass wrongly placed it on nh3-sw1, for two reasons. The UDM's port
    table was read 15 s after link-up, before the UDM refreshed it (it polls, and
    lags about a minute). And nh3-sw1's STP BPDUs, seen on the wire, are flooded
    through the UDM's switch, so they do not locate the cable.
  - ⚠ **Why it had no IP: Linux had the port DOWN.** When `igc` downs the I226-LM
    it powers off the PHY, and AMT's link goes with it (measured: "Link detected:
    no" while down, a 1 Gb link and a DHCP lease within minutes of `ip link set
    up`).
    - It was down because we pulled `enp88s0` out of vmbr0 on 2026-09-25 to
      avoid a loop, which left it unconfigured at boot. Our own change caused it.
    - **Fix, persistent:** `auto enp88s0` / `iface enp88s0 inet manual` in
      `/etc/network/interfaces` (PVE's "Autostart"; it stays up with no address
      and no bridge). `/etc/sysctl.d/90-amt-port.conf` turns IPv6 off on it, so the
      host never SLAACs an address or default route onto nh3-default. Backup:
      `/root/interfaces.bak-20260925-amt`.
    - **Never down this port or remove the `auto` line**, or AMT goes dark.
  - **Not yet verified:** KVM redirection is enabled with User Opt-in = None
    (this needs the AMT admin password), and AMT keeps its link with the host
    powered off (this depends on the MEBx power policy).
  - MEBx menu: Ctrl+P at boot.
- **Console OOB exists: a Sipeed NanoKVM** is attached (USB `3346:1009` on the host;
  web UI **`https://10.100.250.171`**, switch port 23, nh3-mgmt). It gives video and
  keyboard, so BIOS, MEBx and a host that booted without network are all reachable
  remotely. Power control through it is unverified (the MS-01 has no standard ATX
  front-panel header). ⚠ It captures the **iGPU's HDMI**, and RTX-class cards output
  mini-DP only, so keep the BIOS primary display on the iGPU after a GPU install.
- ⚠ **A GPU in the slot RENAMES every NIC.** With a card present the slot's root
  port 00:01.0 takes PCI bus 01 and everything behind it moves down a bus (measured
  on esh-pve): X710 02→03, I226-V 57→58, I226-LM 58→59, Wi-Fi 59→5a. The names
  would become `enp3s0f0np0`, `enp88s0`, `enp89s0`, `wlp90s0`. vmbr0 would then boot
  with no uplink and take the whole NH3 site's VMs off the network.
  **Pinned 2026-09-25** by MAC with `/etc/systemd/network/10-pin-*.link`, also baked
  into every initramfs and synced to the ESP (`playbooks/nh3-pve-pin-nic-names.yaml`,
  verified with `udevadm test-builtin net_setup_link`). If the host ever boots
  without network anyway: open the NanoKVM console, run `ip -br link`, and compare
  names against `/etc/network/interfaces` (backup in `/root/nic-pin-2026-09-25/`).
- **`enp88s0` (the AMT port) is not a vmbr0 bridge port** (removed 2026-09-25).
  STP is off, and UDM port 6 carries the same VLANs as the uplink, so bridging it
  would loop the site LAN. It is held admin-UP on its own instead (AMT bullet above).
- **GPU: RTX 2000E Ada at `01:00.0`** (`10de:28b0`), installed 2026-09-25. The
  NIC pins held: the X710 moved to bus 03 and every NIC kept its name. **NVIDIA
  580.178.04** (open modules, DKMS) has been on the host since 2026-09-25 at 1527
  (`playbooks/pve-nvidia-host.yaml`), with `nvidia-persistenced` ordered before
  `pve-guests`. It serves CT 109 nh3-ml1.
- **iGPU restored (2026-09-25 visit).** With the card in, the BIOS's Auto primary
  display had hidden the iGPU. That left the NanoKVM (iGPU HDMI) blind and would
  have blinded AMT KVM too, since AMT captures only the iGPU. It was set on the
  visit, and since the 1523 boot `00:02.0` is back with `boot_vga=1` and i915
  loaded. The NanoKVM should have video again (not checked from here).
  `systemctl reboot --firmware-setup` works (`OsIndicationsSupported` bit 0), so
  nobody has to catch Del at POST. The AMI `Setup` variables are not
  runtime-visible, so there is no OS-side BIOS patch. esh-pve, the same MS-01 and
  BIOS `AHWSA.1.17`, is the known-good reference.
- **Secure Boot: OFF since the 2026-09-25 visit** (`mokutil --sb-state`: disabled),
  which matches esh-pve. While it was ON (lockdown `integrity`), the DKMS NVIDIA
  module was refused and the first install rolled itself back. The playbook's
  pre-flight refuses if it is ever turned back on without an enrolled DKMS MOK.
- ⚠ **Kernel jumped `6.8.12-11` → `6.8.12-43` at the visit reboot, pulled in by
  our own playbook.** At 1419 the headers step ran `apt-get install
  proxmox-headers-6.8`. That upgraded the `proxmox-kernel-6.8` meta and installed
  `proxmox-kernel-6.8.12-43-pve-signed`. Nobody chose the new kernel; it booted
  because it was the newest.
  Side effect: **every -43 boot oopses in Bluetooth** (`btmtk_usb_hci_wmt_sync` →
  NULL deref in `hci_power_on`, the MS-01's MediaTek BT; taint `D`). It hit on all
  three -43 boots and on none of the -11 boots. esh-pve on `-42` shows the same
  oops and has run fine, so it is benign so far: only the BT worker dies. The fix
  is to blacklist `btusb` on both hypervisors. That is not done, because it only
  takes effect at the next boot.
- **Runs warmer than esh-pve, its twin (Beszel, 2026-09-15 → 09-25).**
  - nh3-pve's CPU package: median of 2-hour averages 76 °C, peak 2-hour average
    88 °C (2026-09-23, CPU ~15%).
  - esh-pve: medians 52–55 °C, peak 20-min average 78 °C.
  - During the 2026-09-25 2100 vzdump, the 1-minute samples reached 90 °C at
    13–21% CPU. They were back to 61 °C once the job finished.
  - Workload is a confound: nh3-pve usually carries more load, so this does not
    prove a cooling fault. Worth checking airflow and dust on the next visit.
  - TjMax is 100 °C. The Beszel CPU alert on this host is set at >95 °C for
    5 minutes.
- **lxc-pve 6.0.0-1 → 6.0.0-2** (2026-09-25 1533, that one package only). This is
  Proxmox fix #7006. Without it, runc 1.5 inside a nesting CT fails every
  `docker run`. `playbooks/gpu-lxc.yaml` now upgrades it first.
- **OOB plan status.** Prime ruled on 2026-09-25 via Miranda to HOLD console-blind
  until the site visit. Target: the NanoKVM moves to pfi-gx10, and this MS-01 uses
  its own AMT. The visit did IGFX, turned SB off and cabled plus enabled AMT.
  AMT came up on the network at 2224 once the host stopped downing its port.
  **Still open:** confirm AMT KVM shows the console. Only then move the NanoKVM to
  the gx10.

## Refresh state

```bash
scripts/refresh-server-info.sh nh3-pve
```

Same Proxmox-inspect caveat as `pfi-pve`: snapshots capture host-level detail only. VM/LXC/storage enumeration needs the Proxmox tooling manually.

## Placement rule

Hypervisor. New NH-site workloads land here as VMs.