From 50f113cd7dc8f72a0ee719423964b03c350fe80e Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 25 Sep 2026 23:08:53 -0700 Subject: [PATCH] feat(nh3-pve): AMT static on nh3-mgmt with KVM enabled and opt-in none; WS-Man helper - AMT re-IP'd over WS-Man to static 10.100.250.61/24 (gw/DNS .1): AMT keeps its old DHCP lease across a VLAN move, so it was unreachable until changed (reached via a temporary on-link /32 on vmbr0 + ssh tunnel, since removed). - KVM redirection enabled, redirection listener on, IPS_OptInService OptInRequired=0; all read back. Password vaulted as nh3-pve/amt-admin. - scripts/amt-wsman.py: stdlib WS-Man get/put/invoke client (digest auth, AMT 16 legacy-renegotiation TLS), password from $AMT_PW. --- persistent-memory.md | 21 +++++------ scripts/amt-wsman.py | 77 +++++++++++++++++++++++++++++++++++++++ servers/nh3-pve/README.md | 34 ++++++++++------- 3 files changed, 107 insertions(+), 25 deletions(-) create mode 100755 scripts/amt-wsman.py diff --git a/persistent-memory.md b/persistent-memory.md index 93fab4c..70b8f5f 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -132,18 +132,15 @@ hermes-gateway, Booth, ttyd, zellij-web and dev-backup are running. `servers/nh3-ml1/README.md` exists. - ⏳ **Prime's call: gateway routing.** Recommendation: load-share `qwen3-embedding` + `reranker` across esh-ml1 and nh3-ml1. -- **AMT → nh3-mgmt at 2237 (Prime): `https://10.100.250.61:16993`** - (`nh3-pve-amt.nh3.internal`, Homepage *NH3-PVE-AMT*). UDM port 6 is native - mgmt with tags blocked, and the reservation is `.61`. - - ⏳ AMT is still sitting on its old `10.100.0.151` lease and is unreachable - until that lease rebinds or expires: ~1920 to ~2224 on 09-26. AMT does not - re-DHCP when its VLAN changes or its link drops. - - A background poll watches `.61`. - - Faster route: the AMT admin password (WS-Man). - - Earlier root cause: Linux had the port down (the `igc` PHY was off). Fixed - with `auto enp88s0` plus IPv6 off and `arp_ignore=8` via sysctl.d. - - **Open:** check KVM + Opt-in None; a dummy HDMI plug before the NanoKVM - moves. +- ✅ **AMT on nh3-mgmt, static `10.100.250.61` (2306),** `nh3-pve-amt.nh3.internal`, + Homepage *NH3-PVE-AMT*. Password in the vault as `nh3-pve/amt-admin`. + - UDM port 6 is native mgmt with tags blocked. + - KVM on, redirection listener on, Opt-in None; set through + `scripts/amt-wsman.py`. + - Root cause earlier tonight: Linux had the port down (the `igc` PHY was off). + Fixed with `auto enp88s0`, IPv6 off and `arp_ignore=8`. + - **Open:** Prime's first MeshCommander session, then a dummy HDMI plug, then + move the NanoKVM to the gx10. - Found and fixed: **lxc-pve 6.0.0-1 broke Docker in the CT** (runc 1.5 sysctl reopen denied). Upgraded that one package to 6.0.0-2 (Proxmox fix #7006), and `gpu-lxc.yaml` now does this itself. diff --git a/scripts/amt-wsman.py b/scripts/amt-wsman.py new file mode 100755 index 0000000..e315587 --- /dev/null +++ b/scripts/amt-wsman.py @@ -0,0 +1,77 @@ +#!/usr/bin/env python3 +"""Minimal Intel AMT WS-Man client, stdlib only (no wsmancli / MeshCommander). + +The password comes from $AMT_PW, which you fill from the vault; it is never +printed: + export AMT_PW="$(secret get nh3-pve/amt-admin)" + scripts/amt-wsman.py 10.100.250.61:16993 get AMT_EthernetPortSettings \ + "InstanceID=Intel(r) AMT Ethernet Port Settings 0" + scripts/amt-wsman.py get IPS_OptInService + scripts/amt-wsman.py put [Selector=Value] + scripts/amt-wsman.py invoke [Selector=Value] + +A Put body is the full instance, fields in the order Get returns them. AMT 16 +needs TLS legacy renegotiation (handled below). Used 2026-09-25 to re-IP nh3-pve's +AMT to static 10.100.250.61, enable KVM (CIM_KVMRedirectionSAP RequestStateChange +2), switch the redirection listener on, and set OptInRequired=0. See +servers/nh3-pve/README.md. +""" +import os, ssl, sys, urllib.request, uuid, re + +AMT = "http://intel.com/wbem/wscim/1/amt-schema/1/" +IPS = "http://intel.com/wbem/wscim/1/ips-schema/1/" +CIM = "http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/" + + +def uri(cls): + return (AMT if cls.startswith("AMT_") else IPS if cls.startswith("IPS_") else CIM) + cls + + +def call(hostport, action, cls, body="", selectors=None, action_uri=None): + sel = "" + if selectors: + sel = "" + "".join(f'{v}' for k, v in selectors.items()) + "" + env = f""" + +
{action_uri or "http://schemas.xmlsoap.org/ws/2004/09/transfer/" + action}/wsman +{uri(cls)}uuid:{uuid.uuid4()} +http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous +PT60S{sel}
{body}""" + ctx = ssl.create_default_context() + ctx.check_hostname = False + ctx.verify_mode = ssl.CERT_NONE + ctx.options |= 0x4 # OP_LEGACY_SERVER_CONNECT: AMT 16 needs legacy renegotiation + ctx.set_ciphers("DEFAULT@SECLEVEL=0") + url = f"https://{hostport}/wsman" + pm = urllib.request.HTTPPasswordMgrWithDefaultRealm() + pm.add_password(None, url, "admin", os.environ["AMT_PW"]) + op = urllib.request.build_opener(urllib.request.HTTPSHandler(context=ctx), urllib.request.HTTPDigestAuthHandler(pm)) + req = urllib.request.Request(url, env.encode(), {"Content-Type": "application/soap+xml;charset=UTF-8"}) + return op.open(req, timeout=60).read().decode() + + +def fields(xml): + m = re.search(r"<(?:\w+:)?Body>(.*)", xml, re.S) + inner = m.group(1) if m else xml + return re.findall(r"<(?:\w+:)?(\w+)>([^<]*)", inner) + + +if __name__ == "__main__": + hp, verb, cls = sys.argv[1:4] + rest = sys.argv[4:] + if verb == "get": + sel = dict(a.split("=", 1) for a in rest) or None + for k, v in fields(call(hp, "Get", cls, selectors=sel)): + print(f"{k}={v}") + elif verb == "invoke": + method, body = rest[0], open(rest[1]).read() + sel = dict(a.split("=", 1) for a in rest[2:]) or None + out = call(hp, None, cls, body=body, selectors=sel, action_uri=uri(cls) + "/" + method) + for k, v in fields(out): + print(f"{k}={v}") + elif verb == "put": + body = open(rest[0]).read() + sel = dict(a.split("=", 1) for a in rest[1:]) or None + out = call(hp, "Put", cls, body=body, selectors=sel) + for k, v in fields(out): + print(f"{k}={v}") diff --git a/servers/nh3-pve/README.md b/servers/nh3-pve/README.md index 1ae55db..a8a97ca 100644 --- a/servers/nh3-pve/README.md +++ b/servers/nh3-pve/README.md @@ -57,19 +57,27 @@ not power back on by itself. without checking which one has carrier (`ip -br link`). The bridge carries the I226-V's MAC `…:96:0d` because it is the first port listed. - **AMT: `https://10.100.250.61:16993`** (`nh3-pve-amt.nh3.internal`; Homepage - card *NH3-PVE-AMT* under Infra - NH3). Intel AMT 16.1.25. It came up at 2224 - on 2026-09-25 as `10.100.0.151` and was moved to nh3-mgmt at 2237 on Prime's - order. - - ⏳ **Until its old lease runs out, AMT still holds `10.100.0.151`**, and - nothing reaches it at either address. AMT does not re-DHCP when its VLAN - changes. - - A link drop, real or `ethtool -r`, makes it send only one ARP for its old - gateway `10.100.0.1`, which gets no answer. It sends no DHCP. - - The lease is 24 h from 2224 on 09-25. Renewal at 12 h goes unicast to - `10.100.0.1`, which is unreachable from mgmt. The rebind at ~21 h - (~1920 on 09-26) or expiry (~2224 on 09-26) should NAK it or start a new - DHCP, and the reservation then gives `.61`. - - The faster route needs the AMT admin password (WS-Man) or the MEBx menu. + card *NH3-PVE-AMT* under Infra - NH3). Intel AMT 16.1.25, **Admin Control + Mode**. User `admin`; the password is in the vault as `nh3-pve/amt-admin`. + - **Static IP since 2026-09-25 2306:** `10.100.250.61/24`, gateway and DNS + `10.100.250.1`. It is set in AMT through WS-Man, and the UDM reservation + stays as a placeholder. + - Why static: AMT does not re-DHCP after a VLAN move. It kept its old + nh3-default lease, and a link drop makes it send only one ARP for its old + gateway. Static also means OOB does not depend on DHCP. + - How it was changed while AMT sat on the old address: a temporary + `10.100.0.250/32` plus a `/32` route on nh3-pve's vmbr0 (same L2), and an + ssh tunnel. The temporary address was removed afterwards. + - **Remote-screen settings, verified by reading them back:** + - KVM enabled (`CIM_KVMRedirectionSAP` EnabledState 6, ready). + - Redirection listener on (`AMT_RedirectionService` 32771, ListenerEnabled). + - **User Opt-in = None** (`IPS_OptInService.OptInRequired` 0), so no 6-digit + code is needed at the rack. + - The VNC port 5900 is off. Use MeshCommander. + - Tool: `scripts/amt-wsman.py`. + - ⚠ **Before the NanoKVM leaves this box, fit a dummy HDMI plug** on the iGPU + HDMI. AMT KVM draws only an active iGPU output. MS-01 owners commonly report + a black KVM screen without one. - It uses a self-signed cert and TLS 1.2 with legacy renegotiation, so OpenSSL 3 clients need `Options = UnsafeLegacyRenegotiation`; browsers cope. 16992 (plain HTTP) is closed. 664 (TLS redirection: SOL/IDER/KVM) is open. It does