173 lines
11 KiB
Markdown
173 lines
11 KiB
Markdown
# nh3-pve
|
||
|
||
Proxmox VE hypervisor for the NH3 site (`nh3-vmhost.phasefinal.com`).
|
||
|
||
## Network
|
||
|
||
- **LAN IP:** 10.100.250.60
|
||
- **FQDN:** `nh3-vmhost.phasefinal.com`
|
||
- **Web UI:** `https://10.100.250.60:8006`
|
||
- **SSH:** `root@nh3-pve` (key auth)
|
||
|
||
## Hardware
|
||
|
||
- **CPU:** 13th Gen Intel Core i9-13900H
|
||
- **RAM:** 62.5 GB
|
||
- **Kernel:** `6.8.12-43-pve` since 2026-09-25 (was `6.8.12-11`; see the kernel bullet below). PVE `8.4.1`, well behind esh-pve's `8.4.20` (177 packages pending)
|
||
- **Storage:** mostly networked — `/mnt/pve/pfi-nh3-nas` (42 TB) mounted from the Synology at `10.100.50.50:/volume1/VMStorage`; ~27 TB used
|
||
|
||
## What it runs
|
||
|
||
Guests as of 2026-09-24 (`qm list` / `pct list` is the live truth). **Expected state
|
||
after a power loss** is the column that matters in a recovery: a guest listed as
|
||
`stays off` is not a casualty.
|
||
|
||
| ID | guest | kind | onboot | expected after power loss |
|
||
|---|---|---|---|---|
|
||
| 100 | nh3-docker (`10.100.50.40`) | VM | 1 | up |
|
||
| 101 | nh3-extdev (`10.100.50.42`) | VM | 1 | up |
|
||
| 102 | nh3-dev (`10.100.10.50`) | VM | 1 | up |
|
||
| 104 | nh3-laser | VM | — | **stays off — on-demand** (Prime, 2026-09-24) |
|
||
| 105 | pbs-nh3 (`10.100.50.90`) | VM | 1 | up (onboot added 2026-09-24; it had none and stayed down after the outage) |
|
||
| 108 | opnsense-lab | VM | 0 | stays off (lab) |
|
||
| 103 | nh3-wg | CT | 1 | up |
|
||
| 106 | nh3-headscale | CT | 1 | up |
|
||
| 107 | nh3-scale | CT | 1 | up (mesh subnet router + fleet egress proxy) |
|
||
| 109 | nh3-ml1 (`10.100.50.80`) | CT | 1 (order 30) | up. GPU LXC, second embed/rerank backend (`servers/nh3-ml1/README.md`); needs the NVIDIA module, so it is the one guest a driver fault can stop |
|
||
|
||
**Power-loss recovery (2026-09-24 outage).** Every guest boots at once, and
|
||
nh3-nas is the slowest to serve NFS. NFS clients now mount nh3-nas shares on
|
||
first access (`playbooks/nh3-nfs-automount.yaml`, applied to nh3-docker and
|
||
nh3-dev), so a late NAS no longer leaves shares down. pbs-nh3's datastore line
|
||
uses `bg` (background retry) instead. `qm guest exec 105` works (the guest agent is
|
||
on), which is the root path into pbs-nh3; infra-ops is not provisioned there.
|
||
**pfi-gx10 (`10.100.50.60`) is bare metal on this site, not a guest here**, and did
|
||
not power back on by itself.
|
||
|
||
## Hardware identity, OOB, and the GPU-slot NIC-rename trap (2026-09-25)
|
||
|
||
- **Box:** Minisforum MS-01 ("Venus Series", BIOS `AHWSA.1.17`): the same model
|
||
and BIOS as esh-pve. i9-13900H, X710 2×SFP+, I226-V + **I226-LM (vPro/AMT)**,
|
||
one PCIe x16 slot (x8 electrical, low-profile, single-slot, slot power only).
|
||
- **Uplink:** one SFP+ 10G port has a cable. Since the 2026-09-25 GPU install it
|
||
is **`enp2s0f1np1`** (MAC `…:96:0c`, PCI 03:00.1). Before that it was
|
||
`enp2s0f0np0` (`…:96:0b`), so the cable moved cages during the visit. The pin
|
||
follows MAC, so this is a physical move and not a rename. Both SFP+ ports are
|
||
vmbr0 members, so either cage works. **Never drop either port from the bridge**
|
||
without checking which one has carrier (`ip -br link`). The bridge carries the
|
||
I226-V's MAC `…:96:0d` because it is the first port listed.
|
||
- **AMT: LIVE at `https://10.100.0.151:16993`** (`nh3-pve-amt.nh3.internal`),
|
||
Intel AMT 16.1.25, since 2026-09-25 2224. It uses a self-signed cert and TLS
|
||
1.2 with legacy renegotiation, so OpenSSL 3 clients need
|
||
`Options = UnsafeLegacyRenegotiation`; browsers cope. 16992 (plain HTTP) is
|
||
closed. 664 (TLS redirection: SOL/IDER/KVM) is open. It does not answer ping.
|
||
- **Port:** the I226-LM `enp88s0` (MAC `58:47:ca:76:96:0e`) is cabled to
|
||
**PFI-UDMSE port 6** (Prime; confirmed from the UDM, which lists the AMT MAC
|
||
on port 6 at 1 Gb). Port 6 is `forward: all`, meaning untagged nh3-default
|
||
plus every tagged VLAN, so AMT (untagged) lands on **nh3-default
|
||
`10.100.0.0/24`**.
|
||
⚠ The first pass wrongly placed it on nh3-sw1, for two reasons. The UDM's port
|
||
table was read 15 s after link-up, before the UDM refreshed it (it polls, and
|
||
lags about a minute). And nh3-sw1's STP BPDUs, seen on the wire, are flooded
|
||
through the UDM's switch, so they do not locate the cable.
|
||
- **Lease:** UDM DHCP reservation `nh3-pve-amt` → `10.100.0.151`.
|
||
- ⚠ **Why it had no IP: Linux had the port DOWN.** When `igc` downs the I226-LM
|
||
it powers off the PHY, and AMT's link goes with it (measured: "Link detected:
|
||
no" while down, a 1 Gb link and a DHCP lease within minutes of `ip link set
|
||
up`).
|
||
- It was down because we pulled `enp88s0` out of vmbr0 on 2026-09-25 to
|
||
avoid a loop, which left it unconfigured at boot. Our own change caused it.
|
||
- **Fix, persistent:** `auto enp88s0` / `iface enp88s0 inet manual` in
|
||
`/etc/network/interfaces` (PVE's "Autostart"; it stays up with no address
|
||
and no bridge). `/etc/sysctl.d/90-amt-port.conf` turns IPv6 off on it, so the
|
||
host never SLAACs an address or default route onto nh3-default. Backup:
|
||
`/root/interfaces.bak-20260925-amt`.
|
||
- **Never down this port or remove the `auto` line**, or AMT goes dark.
|
||
- **Not yet verified:** KVM redirection is enabled with User Opt-in = None
|
||
(this needs the AMT admin password), and AMT keeps its link with the host
|
||
powered off (this depends on the MEBx power policy).
|
||
- Better home, not done (Prime's call): make UDM port 6 native nh3-mgmt
|
||
(VLAN 250) and drop the tagged VLANs, so AMT alone sits on the management
|
||
network. It would need a new reservation and a DNS change. The UDM API key
|
||
can do it.
|
||
- MEBx menu: Ctrl+P at boot.
|
||
- **Console OOB exists: a Sipeed NanoKVM** is attached (USB `3346:1009` on the host;
|
||
web UI **`https://10.100.250.171`**, switch port 23, nh3-mgmt). It gives video and
|
||
keyboard, so BIOS, MEBx and a host that booted without network are all reachable
|
||
remotely. Power control through it is unverified (the MS-01 has no standard ATX
|
||
front-panel header). ⚠ It captures the **iGPU's HDMI**, and RTX-class cards output
|
||
mini-DP only, so keep the BIOS primary display on the iGPU after a GPU install.
|
||
- ⚠ **A GPU in the slot RENAMES every NIC.** With a card present the slot's root
|
||
port 00:01.0 takes PCI bus 01 and everything behind it moves down a bus (measured
|
||
on esh-pve): X710 02→03, I226-V 57→58, I226-LM 58→59, Wi-Fi 59→5a. The names
|
||
would become `enp3s0f0np0`, `enp88s0`, `enp89s0`, `wlp90s0`. vmbr0 would then boot
|
||
with no uplink and take the whole NH3 site's VMs off the network.
|
||
**Pinned 2026-09-25** by MAC with `/etc/systemd/network/10-pin-*.link`, also baked
|
||
into every initramfs and synced to the ESP (`playbooks/nh3-pve-pin-nic-names.yaml`,
|
||
verified with `udevadm test-builtin net_setup_link`). If the host ever boots
|
||
without network anyway: open the NanoKVM console, run `ip -br link`, and compare
|
||
names against `/etc/network/interfaces` (backup in `/root/nic-pin-2026-09-25/`).
|
||
- **`enp88s0` (the AMT port) is not a vmbr0 bridge port** (removed 2026-09-25).
|
||
STP is off, and UDM port 6 carries the same VLANs as the uplink, so bridging it
|
||
would loop the site LAN. It is held admin-UP on its own instead (AMT bullet above).
|
||
- **GPU: RTX 2000E Ada at `01:00.0`** (`10de:28b0`), installed 2026-09-25. The
|
||
NIC pins held: the X710 moved to bus 03 and every NIC kept its name. **NVIDIA
|
||
580.178.04** (open modules, DKMS) has been on the host since 2026-09-25 at 1527
|
||
(`playbooks/pve-nvidia-host.yaml`), with `nvidia-persistenced` ordered before
|
||
`pve-guests`. It serves CT 109 nh3-ml1.
|
||
- **iGPU restored (2026-09-25 visit).** With the card in, the BIOS's Auto primary
|
||
display had hidden the iGPU. That left the NanoKVM (iGPU HDMI) blind and would
|
||
have blinded AMT KVM too, since AMT captures only the iGPU. It was set on the
|
||
visit, and since the 1523 boot `00:02.0` is back with `boot_vga=1` and i915
|
||
loaded. The NanoKVM should have video again (not checked from here).
|
||
`systemctl reboot --firmware-setup` works (`OsIndicationsSupported` bit 0), so
|
||
nobody has to catch Del at POST. The AMI `Setup` variables are not
|
||
runtime-visible, so there is no OS-side BIOS patch. esh-pve, the same MS-01 and
|
||
BIOS `AHWSA.1.17`, is the known-good reference.
|
||
- **Secure Boot: OFF since the 2026-09-25 visit** (`mokutil --sb-state`: disabled),
|
||
which matches esh-pve. While it was ON (lockdown `integrity`), the DKMS NVIDIA
|
||
module was refused and the first install rolled itself back. The playbook's
|
||
pre-flight refuses if it is ever turned back on without an enrolled DKMS MOK.
|
||
- ⚠ **Kernel jumped `6.8.12-11` → `6.8.12-43` at the visit reboot, pulled in by
|
||
our own playbook.** At 1419 the headers step ran `apt-get install
|
||
proxmox-headers-6.8`. That upgraded the `proxmox-kernel-6.8` meta and installed
|
||
`proxmox-kernel-6.8.12-43-pve-signed`. Nobody chose the new kernel; it booted
|
||
because it was the newest.
|
||
Side effect: **every -43 boot oopses in Bluetooth** (`btmtk_usb_hci_wmt_sync` →
|
||
NULL deref in `hci_power_on`, the MS-01's MediaTek BT; taint `D`). It hit on all
|
||
three -43 boots and on none of the -11 boots. esh-pve on `-42` shows the same
|
||
oops and has run fine, so it is benign so far: only the BT worker dies. The fix
|
||
is to blacklist `btusb` on both hypervisors. That is not done, because it only
|
||
takes effect at the next boot.
|
||
- **Runs warmer than esh-pve, its twin (Beszel, 2026-09-15 → 09-25).**
|
||
- nh3-pve's CPU package: median of 2-hour averages 76 °C, peak 2-hour average
|
||
88 °C (2026-09-23, CPU ~15%).
|
||
- esh-pve: medians 52–55 °C, peak 20-min average 78 °C.
|
||
- During the 2026-09-25 2100 vzdump, the 1-minute samples reached 90 °C at
|
||
13–21% CPU. They were back to 61 °C once the job finished.
|
||
- Workload is a confound: nh3-pve usually carries more load, so this does not
|
||
prove a cooling fault. Worth checking airflow and dust on the next visit.
|
||
- TjMax is 100 °C. The Beszel CPU alert on this host is set at >95 °C for
|
||
5 minutes.
|
||
- **lxc-pve 6.0.0-1 → 6.0.0-2** (2026-09-25 1533, that one package only). This is
|
||
Proxmox fix #7006. Without it, runc 1.5 inside a nesting CT fails every
|
||
`docker run`. `playbooks/gpu-lxc.yaml` now upgrades it first.
|
||
- **OOB plan status.** Prime ruled on 2026-09-25 via Miranda to HOLD console-blind
|
||
until the site visit. Target: the NanoKVM moves to pfi-gx10, and this MS-01 uses
|
||
its own AMT. The visit did IGFX, turned SB off and cabled plus enabled AMT.
|
||
AMT came up on the network at 2224 once the host stopped downing its port.
|
||
**Still open:** confirm AMT KVM shows the console. Only then move the NanoKVM to
|
||
the gx10.
|
||
|
||
## Refresh state
|
||
|
||
```bash
|
||
scripts/refresh-server-info.sh nh3-pve
|
||
```
|
||
|
||
Same Proxmox-inspect caveat as `pfi-pve`: snapshots capture host-level detail only. VM/LXC/storage enumeration needs the Proxmox tooling manually.
|
||
|
||
## Placement rule
|
||
|
||
Hypervisor. New NH-site workloads land here as VMs.
|