137 lines
8.0 KiB
Markdown
137 lines
8.0 KiB
Markdown
# nh3-pve
|
||
|
||
Proxmox VE hypervisor for the NH3 site (`nh3-vmhost.phasefinal.com`).
|
||
|
||
## Network
|
||
|
||
- **LAN IP:** 10.100.250.60
|
||
- **FQDN:** `nh3-vmhost.phasefinal.com`
|
||
- **Web UI:** `https://10.100.250.60:8006`
|
||
- **SSH:** `root@nh3-pve` (key auth)
|
||
|
||
## Hardware
|
||
|
||
- **CPU:** 13th Gen Intel Core i9-13900H
|
||
- **RAM:** 62.5 GB
|
||
- **Kernel:** `6.8.12-11-pve` (Proxmox 8.x)
|
||
- **Storage:** mostly networked — `/mnt/pve/pfi-nh3-nas` (42 TB) mounted from the Synology at `10.100.50.50:/volume1/VMStorage`; ~27 TB used
|
||
|
||
## What it runs
|
||
|
||
Guests as of 2026-09-24 (`qm list` / `pct list` is the live truth). **Expected state
|
||
after a power loss** is the column that matters in a recovery: a guest listed as
|
||
`stays off` is not a casualty.
|
||
|
||
| ID | guest | kind | onboot | expected after power loss |
|
||
|---|---|---|---|---|
|
||
| 100 | nh3-docker (`10.100.50.40`) | VM | 1 | up |
|
||
| 101 | nh3-extdev (`10.100.50.42`) | VM | 1 | up |
|
||
| 102 | nh3-dev (`10.100.10.50`) | VM | 1 | up |
|
||
| 104 | nh3-laser | VM | — | **stays off — on-demand** (Prime, 2026-09-24) |
|
||
| 105 | pbs-nh3 (`10.100.50.90`) | VM | 1 | up (onboot added 2026-09-24; it had none and stayed down after the outage) |
|
||
| 108 | opnsense-lab | VM | 0 | stays off (lab) |
|
||
| 103 | nh3-wg | CT | 1 | up |
|
||
| 106 | nh3-headscale | CT | 1 | up |
|
||
| 107 | nh3-scale | CT | 1 | up (mesh subnet router + fleet egress proxy) |
|
||
|
||
**Power-loss recovery (2026-09-24 outage).** Every guest boots at once, and
|
||
nh3-nas is the slowest to serve NFS. NFS clients now mount nh3-nas shares on
|
||
first access (`playbooks/nh3-nfs-automount.yaml`, applied to nh3-docker and
|
||
nh3-dev), so a late NAS no longer leaves shares down. pbs-nh3's datastore line
|
||
uses `bg` (background retry) instead. `qm guest exec 105` works (the guest agent is
|
||
on), which is the root path into pbs-nh3; infra-ops is not provisioned there.
|
||
**pfi-gx10 (`10.100.50.60`) is bare metal on this site, not a guest here**, and did
|
||
not power back on by itself.
|
||
|
||
## Hardware identity, OOB, and the GPU-slot NIC-rename trap (2026-09-25)
|
||
|
||
- **Box:** Minisforum MS-01 ("Venus Series", BIOS `AHWSA.1.17`): the same model
|
||
and BIOS as esh-pve. i9-13900H, X710 2×SFP+, I226-V + **I226-LM (vPro/AMT)**,
|
||
one PCIe x16 slot (x8 electrical, low-profile, single-slot, slot power only).
|
||
- **Uplink:** one SFP+ 10G port has a cable. Since the 2026-09-25 GPU install it
|
||
is **`enp2s0f1np1`** (MAC `…:96:0c`, PCI 03:00.1). Before that it was
|
||
`enp2s0f0np0` (`…:96:0b`), so the cable moved cages during the visit. The pin
|
||
follows MAC, so this is a physical move and not a rename. Both SFP+ ports are
|
||
vmbr0 members, so either cage works. **Never drop either port from the bridge**
|
||
without checking which one has carrier (`ip -br link`). The bridge carries the
|
||
I226-V's MAC `…:96:0d` because it is the first port listed.
|
||
- **AMT: NOT wired.** The AMT-capable I226-LM (`enp88s0`, MAC `58:47:ca:76:96:0e`)
|
||
has no cable. The ME is present (`/dev/mei0`, "AMT SOL Redirection" 00:16.3), but
|
||
MEBx provisioning status is unknown. To wire it: cable the LM port, then at boot
|
||
press Ctrl+P → set the MEBx password, enable manageability, set network (static
|
||
or DHCP), KVM on, User Opt-in = None, activate network access. MEBx can be driven
|
||
remotely through the NanoKVM below.
|
||
- **Console OOB exists: a Sipeed NanoKVM** is attached (USB `3346:1009` on the host;
|
||
web UI **`https://10.100.250.171`**, switch port 23, nh3-mgmt). It gives video and
|
||
keyboard, so BIOS, MEBx and a host that booted without network are all reachable
|
||
remotely. Power control through it is unverified (the MS-01 has no standard ATX
|
||
front-panel header). ⚠ It captures the **iGPU's HDMI**, and RTX-class cards output
|
||
mini-DP only, so keep the BIOS primary display on the iGPU after a GPU install.
|
||
- ⚠ **A GPU in the slot RENAMES every NIC.** With a card present the slot's root
|
||
port 00:01.0 takes PCI bus 01 and everything behind it moves down a bus (measured
|
||
on esh-pve): X710 02→03, I226-V 57→58, I226-LM 58→59, Wi-Fi 59→5a. The names
|
||
would become `enp3s0f0np0`, `enp88s0`, `enp89s0`, `wlp90s0`. vmbr0 would then boot
|
||
with no uplink and take the whole NH3 site's VMs off the network.
|
||
**Pinned 2026-09-25** by MAC with `/etc/systemd/network/10-pin-*.link`, also baked
|
||
into every initramfs and synced to the ESP (`playbooks/nh3-pve-pin-nic-names.yaml`,
|
||
verified with `udevadm test-builtin net_setup_link`). If the host ever boots
|
||
without network anyway: open the NanoKVM console, run `ip -br link`, and compare
|
||
names against `/etc/network/interfaces` (backup in `/root/nic-pin-2026-09-25/`).
|
||
- **`enp88s0` (the AMT port) is no longer a vmbr0 bridge port** (file edited
|
||
2026-09-25, effective next boot). STP is off, so bridging a second cabled uplink
|
||
into the same L2 would loop the site LAN.
|
||
- **GPU installed 2026-09-25: RTX 2000E Ada at `01:00.0`** (`10de:28b0`). The pins
|
||
held: the X710 moved to bus 03 and every NIC kept its name. No NVIDIA driver yet,
|
||
so `nouveau` binds it (`gsp ctor failed: -2` is expected without GSP firmware).
|
||
⚠ **After the install the iGPU is gone from the PCI bus**: `00:02.0` enumerated on
|
||
the 2026-09-24 boot and is absent now, `/dev/dri` does not exist, and the RTX is
|
||
`boot_vga=1`. The BIOS's Auto primary display picked the PCIe card and hid the
|
||
iGPU, so the NanoKVM's iGPU-HDMI capture has no source. Fix: BIOS → Primary
|
||
Display = IGFX (or enable iGPU Multi-Monitor). Reaching the BIOS now needs a
|
||
display on the RTX's mini-DP, for example the NanoKVM through a mini-DP→HDMI
|
||
adapter. Verify with `lspci | grep 00:02.0` and `boot_vga` on `00:02.0`.
|
||
- ⚠ **Secure Boot is ON here** (`mokutil --sb-state`, lockdown `integrity`;
|
||
enrolled MOK = the Proxmox Secure Boot CA). esh-pve, the same model, has it
|
||
OFF. Any DKMS-built module, the NVIDIA driver included, is refused until its
|
||
key is enrolled through MokManager at boot or Secure Boot is turned off in the
|
||
BIOS. Both need the console, and the console is blind (below). The 2026-09-25
|
||
NVIDIA install failed on this and rolled itself back. What it left, all
|
||
harmless: headers `6.8.12-11` plus the series meta, dkms and build-essential;
|
||
nouveau blacklisted and unloaded; the `.run` staged in `/root/nvidia`.
|
||
- **OOB decision (Prime via Miranda, 2026-09-25): HOLD. nh3-pve stays console-blind
|
||
until the next NH3 site visit.** No BIOS change and no reboot until then. The
|
||
accepted risk: a boot without network means a site trip. **Target end state:**
|
||
the NanoKVM moves to pfi-gx10, and this MS-01 gets out-of-band access through its
|
||
own vPro/AMT (cable the I226-LM `enp88s0` and provision MEBx; see the AMT bullet
|
||
above).
|
||
⚠ **Do the BIOS fix on that same visit.** AMT's KVM redirection captures only
|
||
the Intel iGPU's framebuffer, so with the iGPU hidden as it is now, AMT KVM would
|
||
be just as blind as the NanoKVM. Order: set Primary Display = IGFX, provision
|
||
MEBx, confirm AMT KVM shows the console, and only then move the NanoKVM to the
|
||
gx10.
|
||
**Reaching the BIOS** (2026-09-25 facts). The NanoKVM input is HDMI, the RTX has
|
||
mini-DP only, and the adapters on hand are mini-DP→DP only. The paths:
|
||
(1) an **active** mini-DP→HDMI adapter to feed the NanoKVM, since a passive one
|
||
depends on DP++; (2) a DisplayPort monitor plus a USB keyboard at the rack;
|
||
(3) pull the card, boot on the iGPU, set IGFX explicitly (not Auto), then refit
|
||
the card. The NIC pins make a card-out boot safe.
|
||
`systemctl reboot --firmware-setup` works here (`OsIndicationsSupported` bit 0
|
||
set), so nobody has to catch the Del key at POST.
|
||
**An OS-side patch is not possible:** the AMI `Setup`/`SaSetup` variables are not
|
||
runtime-visible here, unlike on the gx10.
|
||
**Known-good reference: esh-pve.** Same MS-01, same BIOS `AHWSA.1.17`, same RTX
|
||
2000E, and its `00:02.0` is present with `boot_vga=1`. The target state works on
|
||
this hardware.
|
||
|
||
## Refresh state
|
||
|
||
```bash
|
||
scripts/refresh-server-info.sh nh3-pve
|
||
```
|
||
|
||
Same Proxmox-inspect caveat as `pfi-pve`: snapshots capture host-level detail only. VM/LXC/storage enumeration needs the Proxmox tooling manually.
|
||
|
||
## Placement rule
|
||
|
||
Hypervisor. New NH-site workloads land here as VMs.
|