feat(nh3-pve): AMT static on nh3-mgmt with KVM enabled and opt-in none; WS-Man helper

- AMT re-IP'd over WS-Man to static 10.100.250.61/24 (gw/DNS .1): AMT keeps its old DHCP lease across a VLAN move, so it was unreachable until changed (reached via a temporary on-link /32 on vmbr0 + ssh tunnel, since removed). - KVM redirection enabled, redirection listener on, IPS_OptInService OptInRequired=0; all read back. Password vaulted as nh3-pve/amt-admin. - scripts/amt-wsman.py: stdlib WS-Man get/put/invoke client (digest auth, AMT 16 legacy-renegotiation TLS), password from $AMT_PW.
2026-09-25 23:08:53 -07:00
parent b3b75c16f4
commit 50f113cd7d
3 changed files with 107 additions and 25 deletions
@@ -132,18 +132,15 @@ hermes-gateway, Booth, ttyd, zellij-web and dev-backup are running.
  `servers/nh3-ml1/README.md` exists.
 - ⏳ **Prime's call: gateway routing.** Recommendation: load-share
  `qwen3-embedding` + `reranker` across esh-ml1 and nh3-ml1.
- **AMT → nh3-mgmt at 2237 (Prime): `https://10.100.250.61:16993`**
-  (`nh3-pve-amt.nh3.internal`, Homepage *NH3-PVE-AMT*). UDM port 6 is native
-  mgmt with tags blocked, and the reservation is `.61`.
-  - ⏳ AMT is still sitting on its old `10.100.0.151` lease and is unreachable
-    until that lease rebinds or expires: ~1920 to ~2224 on 09-26. AMT does not
-    re-DHCP when its VLAN changes or its link drops.
-  - A background poll watches `.61`.
-  - Faster route: the AMT admin password (WS-Man).
-  - Earlier root cause: Linux had the port down (the `igc` PHY was off). Fixed
-    with `auto enp88s0` plus IPv6 off and `arp_ignore=8` via sysctl.d.
-  - **Open:** check KVM + Opt-in None; a dummy HDMI plug before the NanoKVM
-    moves.
+- ✅ **AMT on nh3-mgmt, static `10.100.250.61` (2306),** `nh3-pve-amt.nh3.internal`,
+  Homepage *NH3-PVE-AMT*. Password in the vault as `nh3-pve/amt-admin`.
+  - UDM port 6 is native mgmt with tags blocked.
+  - KVM on, redirection listener on, Opt-in None; set through
+    `scripts/amt-wsman.py`.
+  - Root cause earlier tonight: Linux had the port down (the `igc` PHY was off).
+    Fixed with `auto enp88s0`, IPv6 off and `arp_ignore=8`.
+  - **Open:** Prime's first MeshCommander session, then a dummy HDMI plug, then
+    move the NanoKVM to the gx10.
 - Found and fixed: **lxc-pve 6.0.0-1 broke Docker in the CT** (runc 1.5 sysctl
  reopen denied). Upgraded that one package to 6.0.0-2 (Proxmox fix #7006), and
  `gpu-lxc.yaml` now does this itself.
@@ -0,0 +1,77 @@
+#!/usr/bin/env python3
+"""Minimal Intel AMT WS-Man client, stdlib only (no wsmancli / MeshCommander).
+
+The password comes from $AMT_PW, which you fill from the vault; it is never
+printed:
+  export AMT_PW="$(secret get nh3-pve/amt-admin)"
+  scripts/amt-wsman.py 10.100.250.61:16993 get AMT_EthernetPortSettings \
+      "InstanceID=Intel(r) AMT Ethernet Port Settings 0"
+  scripts/amt-wsman.py <host:port> get IPS_OptInService
+  scripts/amt-wsman.py <host:port> put <Class> <body.xml> [Selector=Value]
+  scripts/amt-wsman.py <host:port> invoke <Class> <Method> <body.xml> [Selector=Value]
+
+A Put body is the full instance, fields in the order Get returns them. AMT 16
+needs TLS legacy renegotiation (handled below). Used 2026-09-25 to re-IP nh3-pve's
+AMT to static 10.100.250.61, enable KVM (CIM_KVMRedirectionSAP RequestStateChange
+2), switch the redirection listener on, and set OptInRequired=0. See
+servers/nh3-pve/README.md.
+"""
+import os, ssl, sys, urllib.request, uuid, re
+
+AMT = "http://intel.com/wbem/wscim/1/amt-schema/1/"
+IPS = "http://intel.com/wbem/wscim/1/ips-schema/1/"
+CIM = "http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/"
+
+
+def uri(cls):
+    return (AMT if cls.startswith("AMT_") else IPS if cls.startswith("IPS_") else CIM) + cls
+
+
+def call(hostport, action, cls, body="", selectors=None, action_uri=None):
+    sel = ""
+    if selectors:
+        sel = "<w:SelectorSet>" + "".join(f'<w:Selector Name="{k}">{v}</w:Selector>' for k, v in selectors.items()) + "</w:SelectorSet>"
+    env = f"""<?xml version="1.0" encoding="utf-8"?>
+<Envelope xmlns="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:w="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd">
+<Header><a:Action>{action_uri or "http://schemas.xmlsoap.org/ws/2004/09/transfer/" + action}</a:Action><a:To>/wsman</a:To>
+<w:ResourceURI>{uri(cls)}</w:ResourceURI><a:MessageID>uuid:{uuid.uuid4()}</a:MessageID>
+<a:ReplyTo><a:Address>http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</a:Address></a:ReplyTo>
+<w:OperationTimeout>PT60S</w:OperationTimeout>{sel}</Header><Body>{body}</Body></Envelope>"""
+    ctx = ssl.create_default_context()
+    ctx.check_hostname = False
+    ctx.verify_mode = ssl.CERT_NONE
+    ctx.options |= 0x4  # OP_LEGACY_SERVER_CONNECT: AMT 16 needs legacy renegotiation
+    ctx.set_ciphers("DEFAULT@SECLEVEL=0")
+    url = f"https://{hostport}/wsman"
+    pm = urllib.request.HTTPPasswordMgrWithDefaultRealm()
+    pm.add_password(None, url, "admin", os.environ["AMT_PW"])
+    op = urllib.request.build_opener(urllib.request.HTTPSHandler(context=ctx), urllib.request.HTTPDigestAuthHandler(pm))
+    req = urllib.request.Request(url, env.encode(), {"Content-Type": "application/soap+xml;charset=UTF-8"})
+    return op.open(req, timeout=60).read().decode()
+
+
+def fields(xml):
+    m = re.search(r"<(?:\w+:)?Body>(.*)</(?:\w+:)?Body>", xml, re.S)
+    inner = m.group(1) if m else xml
+    return re.findall(r"<(?:\w+:)?(\w+)>([^<]*)</(?:\w+:)?\1>", inner)
+
+
+if __name__ == "__main__":
+    hp, verb, cls = sys.argv[1:4]
+    rest = sys.argv[4:]
+    if verb == "get":
+        sel = dict(a.split("=", 1) for a in rest) or None
+        for k, v in fields(call(hp, "Get", cls, selectors=sel)):
+            print(f"{k}={v}")
+    elif verb == "invoke":
+        method, body = rest[0], open(rest[1]).read()
+        sel = dict(a.split("=", 1) for a in rest[2:]) or None
+        out = call(hp, None, cls, body=body, selectors=sel, action_uri=uri(cls) + "/" + method)
+        for k, v in fields(out):
+            print(f"{k}={v}")
+    elif verb == "put":
+        body = open(rest[0]).read()
+        sel = dict(a.split("=", 1) for a in rest[1:]) or None
+        out = call(hp, "Put", cls, body=body, selectors=sel)
+        for k, v in fields(out):
+            print(f"{k}={v}")
@@ -57,19 +57,27 @@ not power back on by itself.
  without checking which one has carrier (`ip -br link`). The bridge carries the
  I226-V's MAC `…:96:0d` because it is the first port listed.
 - **AMT: `https://10.100.250.61:16993`** (`nh3-pve-amt.nh3.internal`; Homepage
-  card *NH3-PVE-AMT* under Infra - NH3). Intel AMT 16.1.25. It came up at 2224
-  on 2026-09-25 as `10.100.0.151` and was moved to nh3-mgmt at 2237 on Prime's
-  order.
-  - ⏳ **Until its old lease runs out, AMT still holds `10.100.0.151`**, and
-    nothing reaches it at either address. AMT does not re-DHCP when its VLAN
-    changes.
-    - A link drop, real or `ethtool -r`, makes it send only one ARP for its old
-      gateway `10.100.0.1`, which gets no answer. It sends no DHCP.
-    - The lease is 24 h from 2224 on 09-25. Renewal at 12 h goes unicast to
-      `10.100.0.1`, which is unreachable from mgmt. The rebind at ~21 h
-      (~1920 on 09-26) or expiry (~2224 on 09-26) should NAK it or start a new
-      DHCP, and the reservation then gives `.61`.
-    - The faster route needs the AMT admin password (WS-Man) or the MEBx menu.
+  card *NH3-PVE-AMT* under Infra - NH3). Intel AMT 16.1.25, **Admin Control
+  Mode**. User `admin`; the password is in the vault as `nh3-pve/amt-admin`.
+  - **Static IP since 2026-09-25 2306:** `10.100.250.61/24`, gateway and DNS
+    `10.100.250.1`. It is set in AMT through WS-Man, and the UDM reservation
+    stays as a placeholder.
+    - Why static: AMT does not re-DHCP after a VLAN move. It kept its old
+      nh3-default lease, and a link drop makes it send only one ARP for its old
+      gateway. Static also means OOB does not depend on DHCP.
+    - How it was changed while AMT sat on the old address: a temporary
+      `10.100.0.250/32` plus a `/32` route on nh3-pve's vmbr0 (same L2), and an
+      ssh tunnel. The temporary address was removed afterwards.
+  - **Remote-screen settings, verified by reading them back:**
+    - KVM enabled (`CIM_KVMRedirectionSAP` EnabledState 6, ready).
+    - Redirection listener on (`AMT_RedirectionService` 32771, ListenerEnabled).
+    - **User Opt-in = None** (`IPS_OptInService.OptInRequired` 0), so no 6-digit
+      code is needed at the rack.
+    - The VNC port 5900 is off. Use MeshCommander.
+    - Tool: `scripts/amt-wsman.py`.
+  - ⚠ **Before the NanoKVM leaves this box, fit a dummy HDMI plug** on the iGPU
+    HDMI. AMT KVM draws only an active iGPU output. MS-01 owners commonly report
+    a black KVM screen without one.
  - It uses a self-signed cert and TLS 1.2 with legacy renegotiation, so OpenSSL 3
    clients need `Options = UnsafeLegacyRenegotiation`; browsers cope. 16992
    (plain HTTP) is closed. 664 (TLS redirection: SOL/IDER/KVM) is open. It does