From 2b49be49721c58f588703f5f06291edb65bd40b0 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 25 Sep 2026 13:34:17 -0700 Subject: [PATCH] =?UTF-8?q?docs(pfi-gx10,nh3-pve):=20AC-restore=20patch=20?= =?UTF-8?q?validated;=20nh3-pve=20OOB=20held=20blind=20until=20next=20NH3?= =?UTF-8?q?=20visit=20(NanoKVM=20=E2=86=92=20gx10,=20MS-01=20=E2=86=92=20o?= =?UTF-8?q?wn=20vPro)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../2026-09-24-gx10-ac-restore-patch.md | 3 +++ persistent-memory.md | 23 ++++++++++--------- servers/nh3-pve/README.md | 11 +++++++++ servers/pfi-gx10/README.md | 14 +++++++---- 4 files changed, 35 insertions(+), 16 deletions(-) diff --git a/persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md b/persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md index dafab5c..69aeeda 100644 --- a/persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md +++ b/persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md @@ -27,3 +27,6 @@ Prime left NH3. **He will pull and restore AC on 2026-09-25.** Comes up alone → works; drop "NOT YET VALIDATED" from the README. Stays off → he presses the button, revert from the backup, and set it in firmware via a PiKVM (he intends to add one with ATX control, which also covers remote power-on regardless). + +**Validated 2026-09-25 (Prime, relayed by Miranda 1333):** Prime replugged AC and +the box came up by itself at 1108:54 PT, with no button press. The patch works. diff --git a/persistent-memory.md b/persistent-memory.md index cda7e4a..fcd1b91 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-25 ~1122 PT (⭐⭐ nh3-pve back from the GPU install, checklist passed; ⚠ iGPU hidden by BIOS → NanoKVM blind, needs on-site BIOS fix. Today: TEI = fleet embed/rerank engine, esh-ml1 sole backend + reward seat moved there, fv-ml1 GPU 1 freed ~15 GB; esh-ml1 monitoring wired; homepage wedge cleared; NIC names pinned on nh3-pve. 34 old entries archived.)_ +_Last updated: 2026-09-25 ~1336 PT (⭐⭐ nh3-pve back from the GPU install, checklist passed; ⚠ iGPU hidden by BIOS → NanoKVM blind, HELD to next NH3 visit (Prime); gx10 AC-restore VALIDATED. Today: TEI = fleet embed/rerank engine, esh-ml1 sole backend + reward seat moved there, fv-ml1 GPU 1 freed ~15 GB; esh-ml1 monitoring wired; homepage wedge cleared; NIC names pinned on nh3-pve. 34 old entries archived.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -123,14 +123,15 @@ _As of 2026-09-24 ~2245 PT._ NFS, the post office (push), svos, hermes-gateway, Booth, talk, ttyd, zellij, backup timers, pbs-nh3 datastore, the mesh (irv-ml1 direct), the egress proxy and NH3 DNS are all fine. Open items: -- ⚠ **iGPU gone → NanoKVM has no picture.** The BIOS Auto primary display picked - the RTX and hid `00:02.0`. It needs a BIOS change on site. See - `servers/nh3-pve/README.md`. +- ⚠ **iGPU gone → NanoKVM has no picture. Prime's call (1333, via Miranda): HOLD, + and stay blind until the next NH3 visit.** No BIOS change and no reboot. Target: + the NanoKVM moves to pfi-gx10, and nh3-pve gets its own vPro/AMT. ⚠ AMT KVM + captures only the iGPU, so the IGFX BIOS fix has to happen on that same visit, + before the NanoKVM leaves. The order is in `servers/nh3-pve/README.md`. - The uplink cable moved from `enp2s0f0np0` to `enp2s0f1np1`. It is harmless because both ports are bridge members. Recorded in the README. -- **pfi-gx10 booted 1108:54** after being off since 09-24 1435. That fits the AC - restore working, but only Prime knows whether he pressed the button. The README - keeps "NOT YET VALIDATED" until he says so. +- ✅ **pfi-gx10 AC-restore patch VALIDATED.** Prime confirmed he replugged AC and + it came up by itself at 1108:54; the README is updated. - Pre-existing and not caused by the reboot: `althing-pump-infra-hermes` and `althing-seat-daemon-jekyll` are both disabled (stopped 09-19 and 09-24). The `infra-hermes` handle shows `pull`. @@ -194,11 +195,11 @@ esh-docker-vm). Down 0613:44–0616:58 PT; homepage needed a manual `docker start` (a failed restart had marked it stopped). Signature + recovery: `servers/esh-docker-vm/README.md`. -### pfi-gx10 is OFF — AC-pull test 2026-09-25 (Prime) +### ✅ pfi-gx10 power-on after AC loss — VALIDATED 2026-09-25 (Prime) -UEFI "Restore AC Power Loss" patch applied, **unvalidated**. Do NOT test it with -a shutdown (stays off by design). Outcomes and revert in -`servers/pfi-gx10/README.md`. → `persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md` +The UEFI patch works: an AC pull and replug brings the box up by itself. A +shutdown is still not a test. Re-check after any BIOS update. Revert details are +in `servers/pfi-gx10/README.md`. → `persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md` ### Still open from 09-22 diff --git a/servers/nh3-pve/README.md b/servers/nh3-pve/README.md index 943996a..b0f475b 100644 --- a/servers/nh3-pve/README.md +++ b/servers/nh3-pve/README.md @@ -90,6 +90,17 @@ not power back on by itself. Display = IGFX (or enable iGPU Multi-Monitor). Reaching the BIOS now needs a display on the RTX's mini-DP, for example the NanoKVM through a mini-DP→HDMI adapter. Verify with `lspci | grep 00:02.0` and `boot_vga` on `00:02.0`. +- **OOB decision (Prime via Miranda, 2026-09-25): HOLD. nh3-pve stays console-blind + until the next NH3 site visit.** No BIOS change and no reboot until then. The + accepted risk: a boot without network means a site trip. **Target end state:** + the NanoKVM moves to pfi-gx10, and this MS-01 gets out-of-band access through its + own vPro/AMT (cable the I226-LM `enp88s0` and provision MEBx; see the AMT bullet + above). + ⚠ **Do the BIOS fix on that same visit.** AMT's KVM redirection captures only + the Intel iGPU's framebuffer, so with the iGPU hidden as it is now, AMT KVM would + be just as blind as the NanoKVM. Order: while the NanoKVM is still here (on the + RTX through an adapter), set Primary Display = IGFX, provision MEBx, confirm AMT + KVM shows the console, and only then move the NanoKVM to the gx10. ## Refresh state diff --git a/servers/pfi-gx10/README.md b/servers/pfi-gx10/README.md index 13adbbb..533ea3d 100644 --- a/servers/pfi-gx10/README.md +++ b/servers/pfi-gx10/README.md @@ -54,7 +54,7 @@ trivially while the desktop is still running. The playbook will not stop GDM while someone holds a seat session. Override with `--var force_dm_stop=true`, or just let the rack-install reboot handle it. -## Power-on after AC loss — patched, NOT YET VALIDATED (2026-09-24) +## Power-on after AC loss — patched 2026-09-24, VALIDATED 2026-09-25 There is **no BMC**, and the ASUS firmware ships with "Restore AC Power Loss" set to stay off, so the box did not come back after the 2026-09-24 NH3 power outage. @@ -80,10 +80,14 @@ are not on this board, and trusting that claim cost two trips to the power butto (infra-ops, 1421 and 1435 on 2026-09-24). **The only valid test is pulling and restoring AC power without touching the button.** -**Status at end of 2026-09-24:** the box is OFF (the 1435 shutdown), with the patch -in place. Prime will pull and restore AC on 2026-09-25. It comes up on its own → -the patch works, so delete "NOT YET VALIDATED" above. It stays off → press the -button, then revert with the backup and set it in the firmware menu via a PiKVM. +**Validated 2026-09-25 (Prime, relayed by Miranda):** the box had been off since +the 09-24 1435 shutdown. Prime pulled and replugged AC, and it **came up by itself**, +booting at 1108:54 PT without the button being pressed. The patch works; keep the +backup, since a firmware update may reset `Setup`. Re-check after any BIOS update +the same way, by pulling AC and not by a shutdown. + +**Planned OOB (Prime, 2026-09-25):** nh3-pve's NanoKVM moves here on the next NH3 +site visit. That also gives this box its first console. See `servers/nh3-pve/README.md`. ## Training — run 3c is staged and ready