fix(nh3-pve): AMT reachable — hold the vPro port up; DHCP reservation + DNS
AMT had no IP because Linux had enp88s0 admin-down (it left vmbr0 on 09-25 and had no auto stanza); igc powers off the I226-LM PHY when down, so AMT had no link. With the port up (no IP, unbridged) AMT took a lease within minutes. - nh3-pve: `auto enp88s0` (manual, unbridged) and sysctl.d disable_ipv6 on it; backup /root/interfaces.bak-20260925-amt; runtime already matched, no reload. - UDM: reservation nh3-pve-amt -> 10.100.0.151 (untagged nh3-default; the cable is on an nh3-sw1 trunk, not the UDM). - DNS nh3-pve-amt.nh3.internal. AMT 16.1.25 answers TLS :16993 and :664.
This commit is contained in:
@@ -82,6 +82,7 @@ hosts:
|
|||||||
- {name: nh3-extdev, site: nh3, v4: 10.100.50.42, note: manager / external-dev box}
|
- {name: nh3-extdev, site: nh3, v4: 10.100.50.42, note: manager / external-dev box}
|
||||||
- {name: nh3-nas, site: nh3, v4: 10.100.50.50, note: Synology RS2418+}
|
- {name: nh3-nas, site: nh3, v4: 10.100.50.50, note: Synology RS2418+}
|
||||||
- {name: nh3-pve, site: nh3, v4: 10.100.250.60, note: Proxmox hypervisor}
|
- {name: nh3-pve, site: nh3, v4: 10.100.250.60, note: Proxmox hypervisor}
|
||||||
|
- {name: nh3-pve-amt, site: nh3, v4: 10.100.0.151, note: Intel AMT (vPro) on nh3-pve's I226-LM — UDM DHCP reservation, untagged nh3-default; web/WS-Man TLS :16993, redirection :664}
|
||||||
- {name: pbs-nh3, site: nh3, v4: 10.100.50.90, note: Proxmox Backup Server — DR mirror}
|
- {name: pbs-nh3, site: nh3, v4: 10.100.50.90, note: Proxmox Backup Server — DR mirror}
|
||||||
- {name: nh3-gw, site: nh3, v4: 10.100.0.1, note: UniFi UDM Pro SE — gateway + controller}
|
- {name: nh3-gw, site: nh3, v4: 10.100.0.1, note: UniFi UDM Pro SE — gateway + controller}
|
||||||
# Irvine is not its own DNS site: it is a single box (irv-ml1) reached over
|
# Irvine is not its own DNS site: it is a single box (irv-ml1) reached over
|
||||||
|
|||||||
@@ -132,11 +132,14 @@ hermes-gateway, Booth, ttyd, zellij-web and dev-backup are running.
|
|||||||
`servers/nh3-ml1/README.md` exists.
|
`servers/nh3-ml1/README.md` exists.
|
||||||
- ⏳ **Prime's call: gateway routing.** Recommendation: load-share
|
- ⏳ **Prime's call: gateway routing.** Recommendation: load-share
|
||||||
`qwen3-embedding` + `reranker` across esh-ml1 and nh3-ml1.
|
`qwen3-embedding` + `reranker` across esh-ml1 and nh3-ml1.
|
||||||
- ⚠ **AMT is cabled (1 Gb link) and reported enabled, but not on the network.** No
|
- ✅ **AMT LIVE 2224: `https://10.100.0.151:16993`** (`nh3-pve-amt.nh3.internal`,
|
||||||
UDM lease for `…:96:0e`, and no 16992/16993 on the four NH3 /24s (the sweep was
|
UDM reservation). It had no IP because Linux had the port DOWN. `igc` powers
|
||||||
positive-controlled). It is not on the USW Pro 24, so probably nh3-sw1. Asked
|
off the PHY and AMT loses its link. Our own 09-25 removal of `enp88s0` from
|
||||||
Prime for the MEBx network settings and which switch port. **The NanoKVM stays
|
vmbr0 caused that. Fix: `auto enp88s0` (up, no IP, unbridged) plus IPv6 off via
|
||||||
on nh3-pve until AMT KVM is confirmed.**
|
sysctl.d, persistent. The cable is on an **nh3-sw1 trunk** (untagged
|
||||||
|
nh3-default), not the UDM.
|
||||||
|
**Open:** check that KVM + Opt-in None are set (needs the AMT admin password),
|
||||||
|
then move the NanoKVM to the gx10.
|
||||||
- Found and fixed: **lxc-pve 6.0.0-1 broke Docker in the CT** (runc 1.5 sysctl
|
- Found and fixed: **lxc-pve 6.0.0-1 broke Docker in the CT** (runc 1.5 sysctl
|
||||||
reopen denied). Upgraded that one package to 6.0.0-2 (Proxmox fix #7006), and
|
reopen denied). Upgraded that one package to 6.0.0-2 (Proxmox fix #7006), and
|
||||||
`gpu-lxc.yaml` now does this itself.
|
`gpu-lxc.yaml` now does this itself.
|
||||||
|
|||||||
+34
-17
@@ -56,19 +56,34 @@ not power back on by itself.
|
|||||||
vmbr0 members, so either cage works. **Never drop either port from the bridge**
|
vmbr0 members, so either cage works. **Never drop either port from the bridge**
|
||||||
without checking which one has carrier (`ip -br link`). The bridge carries the
|
without checking which one has carrier (`ip -br link`). The bridge carries the
|
||||||
I226-V's MAC `…:96:0d` because it is the first port listed.
|
I226-V's MAC `…:96:0d` because it is the first port listed.
|
||||||
- **AMT: cabled and enabled, but NOT reachable (2026-09-25 visit).** The I226-LM
|
- **AMT: LIVE at `https://10.100.0.151:16993`** (`nh3-pve-amt.nh3.internal`),
|
||||||
(`enp88s0`, MAC `58:47:ca:76:96:0e`) now has a 1 Gb link (measured at 1552 by
|
Intel AMT 16.1.25, since 2026-09-25 2224. It uses a self-signed cert and TLS
|
||||||
bringing the port up unbridged for a few seconds). Prime reports AMT enabled in
|
1.2 with legacy renegotiation, so OpenSSL 3 clients need
|
||||||
MEBx. **Nothing answers on the network, though.** The UDM has no client or lease
|
`Options = UnsafeLegacyRenegotiation`; browsers cope. 16992 (plain HTTP) is
|
||||||
for `…:96:0e`, and no host on `10.100.{0,10,50,250}.0/24` has 16992 or 16993
|
closed. 664 (TLS redirection: SOL/IDER/KVM) is open. It does not answer ping.
|
||||||
open. The sweep was checked against known-open ports and does see them. The
|
- **Port:** the I226-LM `enp88s0` (MAC `58:47:ca:76:96:0e`) is cabled to a
|
||||||
cable is not on the USW Pro 24, whose up ports are 19, 22, 23 and 26, all
|
**trunk port on nh3-sw1**, not the UDM (seen from the host: STP from
|
||||||
accounted for. So it is on another switch, most likely nh3-sw1
|
`48:a9:8a:40:8a:51`, untagged nh3-default plus tagged VLANs 10/50/90/250).
|
||||||
(`10.100.250.2`, no infra-ops access). Likely causes: MEBx "Activate Network
|
AMT is untagged, so it lands on **nh3-default `10.100.0.0/24`**.
|
||||||
Access" was not done, a static IP outside those subnets, or a switch port on a
|
- **Lease:** UDM DHCP reservation `nh3-pve-amt` → `10.100.0.151`.
|
||||||
VLAN that gets no DHCP. MEBx menu: Ctrl+P at boot → password, manageability on,
|
- ⚠ **Why it had no IP: Linux had the port DOWN.** When `igc` downs the I226-LM
|
||||||
network (static or DHCP), KVM on, User Opt-in = None, activate network access.
|
it powers off the PHY, and AMT's link goes with it (measured: "Link detected:
|
||||||
**Keep the NanoKVM here until AMT KVM is confirmed.**
|
no" while down, a 1 Gb link and a DHCP lease within minutes of `ip link set
|
||||||
|
up`).
|
||||||
|
- It was down because we pulled `enp88s0` out of vmbr0 on 2026-09-25 to
|
||||||
|
avoid a loop, which left it unconfigured at boot. Our own change caused it.
|
||||||
|
- **Fix, persistent:** `auto enp88s0` / `iface enp88s0 inet manual` in
|
||||||
|
`/etc/network/interfaces` (PVE's "Autostart"; it stays up with no address
|
||||||
|
and no bridge). `/etc/sysctl.d/90-amt-port.conf` turns IPv6 off on it, so the
|
||||||
|
host never SLAACs an address or default route onto nh3-default. Backup:
|
||||||
|
`/root/interfaces.bak-20260925-amt`.
|
||||||
|
- **Never down this port or remove the `auto` line**, or AMT goes dark.
|
||||||
|
- **Not yet verified:** KVM redirection is enabled with User Opt-in = None
|
||||||
|
(this needs the AMT admin password), and AMT keeps its link with the host
|
||||||
|
powered off (this depends on the MEBx power policy).
|
||||||
|
- Better home, not done: nh3-mgmt (VLAN 250), by setting that nh3-sw1 port's
|
||||||
|
native VLAN to 250. infra-ops has no nh3-sw1 access.
|
||||||
|
- MEBx menu: Ctrl+P at boot.
|
||||||
- **Console OOB exists: a Sipeed NanoKVM** is attached (USB `3346:1009` on the host;
|
- **Console OOB exists: a Sipeed NanoKVM** is attached (USB `3346:1009` on the host;
|
||||||
web UI **`https://10.100.250.171`**, switch port 23, nh3-mgmt). It gives video and
|
web UI **`https://10.100.250.171`**, switch port 23, nh3-mgmt). It gives video and
|
||||||
keyboard, so BIOS, MEBx and a host that booted without network are all reachable
|
keyboard, so BIOS, MEBx and a host that booted without network are all reachable
|
||||||
@@ -85,9 +100,9 @@ not power back on by itself.
|
|||||||
verified with `udevadm test-builtin net_setup_link`). If the host ever boots
|
verified with `udevadm test-builtin net_setup_link`). If the host ever boots
|
||||||
without network anyway: open the NanoKVM console, run `ip -br link`, and compare
|
without network anyway: open the NanoKVM console, run `ip -br link`, and compare
|
||||||
names against `/etc/network/interfaces` (backup in `/root/nic-pin-2026-09-25/`).
|
names against `/etc/network/interfaces` (backup in `/root/nic-pin-2026-09-25/`).
|
||||||
- **`enp88s0` (the AMT port) is no longer a vmbr0 bridge port** (file edited
|
- **`enp88s0` (the AMT port) is not a vmbr0 bridge port** (removed 2026-09-25).
|
||||||
2026-09-25, effective next boot). STP is off, so bridging a second cabled uplink
|
STP is off, and that port is on the same nh3-sw1 trunk, so bridging it would
|
||||||
into the same L2 would loop the site LAN.
|
loop the site LAN. It is held admin-UP on its own instead (AMT bullet above).
|
||||||
- **GPU: RTX 2000E Ada at `01:00.0`** (`10de:28b0`), installed 2026-09-25. The
|
- **GPU: RTX 2000E Ada at `01:00.0`** (`10de:28b0`), installed 2026-09-25. The
|
||||||
NIC pins held: the X710 moved to bus 03 and every NIC kept its name. **NVIDIA
|
NIC pins held: the X710 moved to bus 03 and every NIC kept its name. **NVIDIA
|
||||||
580.178.04** (open modules, DKMS) has been on the host since 2026-09-25 at 1527
|
580.178.04** (open modules, DKMS) has been on the host since 2026-09-25 at 1527
|
||||||
@@ -133,7 +148,9 @@ not power back on by itself.
|
|||||||
- **OOB plan status.** Prime ruled on 2026-09-25 via Miranda to HOLD console-blind
|
- **OOB plan status.** Prime ruled on 2026-09-25 via Miranda to HOLD console-blind
|
||||||
until the site visit. Target: the NanoKVM moves to pfi-gx10, and this MS-01 uses
|
until the site visit. Target: the NanoKVM moves to pfi-gx10, and this MS-01 uses
|
||||||
its own AMT. The visit did IGFX, turned SB off and cabled plus enabled AMT.
|
its own AMT. The visit did IGFX, turned SB off and cabled plus enabled AMT.
|
||||||
**Still open:** AMT is not on the network (above), so the NanoKVM stays here.
|
AMT came up on the network at 2224 once the host stopped downing its port.
|
||||||
|
**Still open:** confirm AMT KVM shows the console. Only then move the NanoKVM to
|
||||||
|
the gx10.
|
||||||
|
|
||||||
## Refresh state
|
## Refresh state
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user