From 2bf418c510bf415fae2e3e93f81d4174ee18c3bd Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 25 Sep 2026 11:22:00 -0700 Subject: [PATCH] =?UTF-8?q?docs(nh3-pve):=20record=20GPU-install=20post-bo?= =?UTF-8?q?ot=20state=20=E2=80=94=20RTX=202000E=20at=2001:00.0,=20iGPU=20h?= =?UTF-8?q?idden=20by=20BIOS,=20uplink=20moved=20to=20enp2s0f1np1?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- persistent-memory.md | 23 +++++++++++++++++++---- servers/nh3-pve/README.md | 17 ++++++++++++++++- 2 files changed, 35 insertions(+), 5 deletions(-) diff --git a/persistent-memory.md b/persistent-memory.md index 006c7ff..cda7e4a 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-25 ~1105 PT (⭐⭐ nh3-pve going DOWN for a GPU install, Prime on site; this session ends with nh3-dev — post-boot checklist below. Today: TEI = fleet embed/rerank engine, esh-ml1 sole backend + reward seat moved there, fv-ml1 GPU 1 freed ~15 GB; esh-ml1 monitoring wired; homepage wedge cleared; NIC names pinned on nh3-pve. 34 old entries archived.)_ +_Last updated: 2026-09-25 ~1122 PT (⭐⭐ nh3-pve back from the GPU install, checklist passed; ⚠ iGPU hidden by BIOS → NanoKVM blind, needs on-site BIOS fix. Today: TEI = fleet embed/rerank engine, esh-ml1 sole backend + reward seat moved there, fv-ml1 GPU 1 freed ~15 GB; esh-ml1 monitoring wired; homepage wedge cleared; NIC names pinned on nh3-pve. 34 old entries archived.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -117,10 +117,25 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-09-24 ~2245 PT._ -### ⭐⭐ IN PROGRESS 2026-09-25 ~1100 PT: nh3-pve shut down for a GPU install (Prime on site) +### ⭐⭐ nh3-pve GPU install — host back 2026-09-25 1116, checklist run 1122 -This session runs ON nh3-dev and dies with it. **Successor: work this list -first.** (No /tmp handoff: nh3-dev wipes /tmp at boot.) +**Result:** checklist items 1 and 3–6 pass, and nothing needed restarting. Guests, +NFS, the post office (push), svos, hermes-gateway, Booth, talk, ttyd, zellij, +backup timers, pbs-nh3 datastore, the mesh (irv-ml1 direct), the egress proxy and +NH3 DNS are all fine. Open items: +- ⚠ **iGPU gone → NanoKVM has no picture.** The BIOS Auto primary display picked + the RTX and hid `00:02.0`. It needs a BIOS change on site. See + `servers/nh3-pve/README.md`. +- The uplink cable moved from `enp2s0f0np0` to `enp2s0f1np1`. It is harmless + because both ports are bridge members. Recorded in the README. +- **pfi-gx10 booted 1108:54** after being off since 09-24 1435. That fits the AC + restore working, but only Prime knows whether he pressed the button. The README + keeps "NOT YET VALIDATED" until he says so. +- Pre-existing and not caused by the reboot: `althing-pump-infra-hermes` and + `althing-seat-daemon-jekyll` are both disabled (stopped 09-19 and 09-24). The + `infra-hermes` handle shows `pull`. + +The pre-shutdown record follows, kept for the next GPU visit: Pre-flight done before shutdown: - NIC names pinned by MAC (a GPU renames every NIC on this MS-01), initramfs diff --git a/servers/nh3-pve/README.md b/servers/nh3-pve/README.md index fed832a..943996a 100644 --- a/servers/nh3-pve/README.md +++ b/servers/nh3-pve/README.md @@ -48,7 +48,12 @@ not power back on by itself. - **Box:** Minisforum MS-01 ("Venus Series", BIOS `AHWSA.1.17`): the same model and BIOS as esh-pve. i9-13900H, X710 2×SFP+, I226-V + **I226-LM (vPro/AMT)**, one PCIe x16 slot (x8 electrical, low-profile, single-slot, slot power only). -- **Uplink:** only `enp2s0f0np0` (SFP+ 10G) has a cable. The bridge carries the +- **Uplink:** one SFP+ 10G port has a cable. Since the 2026-09-25 GPU install it + is **`enp2s0f1np1`** (MAC `…:96:0c`, PCI 03:00.1). Before that it was + `enp2s0f0np0` (`…:96:0b`), so the cable moved cages during the visit. The pin + follows MAC, so this is a physical move and not a rename. Both SFP+ ports are + vmbr0 members, so either cage works. **Never drop either port from the bridge** + without checking which one has carrier (`ip -br link`). The bridge carries the I226-V's MAC `…:96:0d` because it is the first port listed. - **AMT: NOT wired.** The AMT-capable I226-LM (`enp88s0`, MAC `58:47:ca:76:96:0e`) has no cable. The ME is present (`/dev/mei0`, "AMT SOL Redirection" 00:16.3), but @@ -75,6 +80,16 @@ not power back on by itself. - **`enp88s0` (the AMT port) is no longer a vmbr0 bridge port** (file edited 2026-09-25, effective next boot). STP is off, so bridging a second cabled uplink into the same L2 would loop the site LAN. +- **GPU installed 2026-09-25: RTX 2000E Ada at `01:00.0`** (`10de:28b0`). The pins + held: the X710 moved to bus 03 and every NIC kept its name. No NVIDIA driver yet, + so `nouveau` binds it (`gsp ctor failed: -2` is expected without GSP firmware). + ⚠ **After the install the iGPU is gone from the PCI bus**: `00:02.0` enumerated on + the 2026-09-24 boot and is absent now, `/dev/dri` does not exist, and the RTX is + `boot_vga=1`. The BIOS's Auto primary display picked the PCIe card and hid the + iGPU, so the NanoKVM's iGPU-HDMI capture has no source. Fix: BIOS → Primary + Display = IGFX (or enable iGPU Multi-Monitor). Reaching the BIOS now needs a + display on the RTX's mini-DP, for example the NanoKVM through a mini-DP→HDMI + adapter. Verify with `lspci | grep 00:02.0` and `boot_vga` on `00:02.0`. ## Refresh state