From 1a2d763de4bee7d867b6445126a3548280c5603b Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 2 Oct 2026 09:06:24 -0700 Subject: [PATCH] ops(nh3-pve): AMT phones home to MeshCentral (CIRA); AMT on DHCP; LAN management now dark by design --- persistent-memory.md | 2 +- servers/nh3-pve/README.md | 13 ++++++++++++- servers/nh3-pve/amt-ethernet-static-revert.xml | 1 + servers/pfi-tacticalrmm/README.md | 10 +++++++--- 4 files changed, 21 insertions(+), 5 deletions(-) create mode 100644 servers/nh3-pve/amt-ethernet-static-revert.xml diff --git a/persistent-memory.md b/persistent-memory.md index aaca328..47ee0b7 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -288,7 +288,7 @@ _As of 2026-10-01 ~0446 PT._ ## Recent decisions - `[2026-10-02]` **Demo outage ~13 min, ROLLED BACK (worldtree-dev URGENT 6684).** Their b193 release commit c2d87263 swept 60 staged deletions into the tree, so the demo api crash-looped. I recreated `worldtree-api` only (`compose up -d --no-deps`) on the `.env`-pinned fa8bc51cc064 (compose.yaml identical at both shas): healthy in 35 s at 15:16Z. ⚠ Their deploy health gate does NOT restore the old container on failure (it only withholds `:latest`); flagged to them. Fix-forward 7ab6ae40 (tag v1.0.0b193 moved onto it) deployed via CI 15:22Z and verified healthy, same seven retired-key WARNINGs; the deploy-gate gap is Worldtree #423. -- `[2026-10-02]` **nh3-pve's AMT is in TacticalRMM's MeshCentral (group `PFI-AMT`, device `nh3-pve-amt`).** The adds had failed silently because TRMM installs MeshCentral `WANonly` (meshuser.js:2682 drops AMT adds). Prime ruled hybrid; switched 0812, 14/14 agents back, AMT connected at once (16.1.25, TLS fine). ⚠ Still reached via nh3-scale ON nh3-pve, so useless when nh3-pve is down; the fix is the IPsec route or CIRA (MPS :4433 is live). 2FA still not forced. Prime's MeshCentral login token is vaulted `pfi-tacticalrmm/meshcentral-login-token`. → `servers/pfi-tacticalrmm/README.md` +- `[2026-10-02]` **nh3-pve's AMT is in TacticalRMM's MeshCentral (group `PFI-AMT`, device `nh3-pve-amt`).** The adds had failed silently because TRMM installs MeshCentral `WANonly` (meshuser.js:2682 drops AMT adds). Prime ruled hybrid; switched 0812, 14/14 agents back, AMT connected at once (16.1.25, TLS fine). **UPDATE 0859: it now PHONES HOME (CIRA)** — MeshCentral mpsPass, ana-gw VIP/policy 76 on 4433, AMT settings via `scripts/amt-cira-setup.py`, AMT moved static → DHCP (Intel: CIRA needs DHCP; reservation keeps .61). Tunnel is independent of nh3-pve. ⚠ While phoning home the AMT REFUSES LAN management (:16993 dark), so manage it via MeshCentral; revert body `servers/nh3-pve/amt-ethernet-static-revert.xml`. 2FA still not forced. Prime's MeshCentral login token is vaulted `pfi-tacticalrmm/meshcentral-login-token`. → `servers/pfi-tacticalrmm/README.md` - `[2026-10-02]` **Prime: dev-backup gets dailies + weeklies — DONE.** Retention is now 48 hourly + newest of 30 days + newest of 12 ISO weeks (`retention.py` beside the script; second path guard on the NAS side; unit fails if kept ≠ expected). Live run 0739: deleted 1, 0 errors, 48 kept = expected. History before 09-30 was already gone; dailies accumulate from today. - `[2026-10-02]` **Prime: MS-03s get Proxmox; dummy plugs in hand.** Still open: what the second MS-03 is for. On arrival: check the NIC chipset (I226-LM = keep the AMT port admin-UP), fit a plug on each, then the parked AMT follow-ups. - `[2026-10-02]` **nh3-dev root grown 250 → 378 GB (Prime resized scsi0; I grew the guest online, no reboot).** Root 372 GB, 58%, 150 GB free, after the 85% alert fired twice in 14 h (uv cache + agent venvs). Swap moved to a 4 GB `/swapfile` (the old `sda5` blocked growth), `RESUME=none`, all initrds rebuilt (`playbooks/nh3-dev-grow-root.yaml`). ⚠ **TODO: delete VM snapshot `pre-rootgrow-20261002` on nh3-pve after the next NATURAL reboot (Prime 2026-10-02: no test reboot).** Trigger: `uptime -s` on nh3-dev later than 2026-10-02 0733. Then check the boot was clean (`swapon --show` = /swapfile; `systemd-analyze` shows no ~30 s stall; `journalctl -b | grep -i resume` has no 'waiting for resume device'), and only then `qm delsnapshot 102 pre-rootgrow-20261002`. diff --git a/servers/nh3-pve/README.md b/servers/nh3-pve/README.md index a8a97ca..dda2866 100644 --- a/servers/nh3-pve/README.md +++ b/servers/nh3-pve/README.md @@ -56,7 +56,18 @@ not power back on by itself. vmbr0 members, so either cage works. **Never drop either port from the bridge** without checking which one has carrier (`ip -br link`). The bridge carries the I226-V's MAC `…:96:0d` because it is the first port listed. -- **AMT: `https://10.100.250.61:16993`** (`nh3-pve-amt.nh3.internal`; Homepage +- ⭐ **AMT PHONES HOME since 2026-10-02 0859 (Prime): use MeshCentral, not the LAN address.** The AMT holds a + CIRA tunnel to `rmm-mesh.phasefinal.com:4433` (TacticalRMM's MeshCentral, group `PFI-AMT`, device + `nh3-pve-amt`), which does not depend on nh3-pve or nh3-scale. MeshCentral logs in over the tunnel + (AMT 16.1.25, power state read). Set up with `scripts/amt-cira-setup.py`; see + `servers/pfi-tacticalrmm/README.md`. + - ⚠ **The AMT is on DHCP now** (UDM reservation keeps `10.100.250.61`). Intel: CIRA does not work on a static IP. + - ⚠ **LAN management is OFF while it is phoning home.** Once AMT decides it is "outside" (random + environment-detection domain `21cade3bec8c`), `10.100.250.61:16993` stops answering, so + `scripts/amt-wsman.py` from nh3-dev no longer reaches it. Manage it through MeshCentral; last resort + is MEBx (Ctrl+P) through the NanoKVM. To go back to static, Put `amt-ethernet-static-revert.xml` + (this dir) on `AMT_EthernetPortSettings` through MeshCentral or from MEBx. +- **AMT (LAN, before 2026-10-02): `https://10.100.250.61:16993`** (`nh3-pve-amt.nh3.internal`; Homepage card *NH3-PVE-AMT* under Infra - NH3). Intel AMT 16.1.25, **Admin Control Mode**. User `admin`; the password is in the vault as `nh3-pve/amt-admin`. - **Static IP since 2026-09-25 2306:** `10.100.250.61/24`, gateway and DNS diff --git a/servers/nh3-pve/amt-ethernet-static-revert.xml b/servers/nh3-pve/amt-ethernet-static-revert.xml new file mode 100644 index 0000000..b185b71 --- /dev/null +++ b/servers/nh3-pve/amt-ethernet-static-revert.xml @@ -0,0 +1 @@ +false10.100.250.1Intel(r) AMT Ethernet Port Settings10.100.250.61Intel(r) AMT Ethernet Port Settings 0falsetrue11658-47-ca-76-96-0e1110.100.250.1truetruefalse255.255.255.0 diff --git a/servers/pfi-tacticalrmm/README.md b/servers/pfi-tacticalrmm/README.md index 0bed6b8..eda4599 100644 --- a/servers/pfi-tacticalrmm/README.md +++ b/servers/pfi-tacticalrmm/README.md @@ -33,9 +33,13 @@ Monitors and manages endpoints, pushes patches, runs scripts, etc. `pfi-tacticalrmm/meshcentral-mpspass`; without it the MPS checked only the 16-char username). FortiGate ana-gw: service `MeshCentral-MPS-4433`, VIP `mps-to-tacticalrmm` (38.120.12.46:4433 → 10.250.50.57) and policy 76 (wan1→servers, accept) — 4433 verified open from the internet, 4434 closed as a control. The AMT - side is `scripts/amt-cira-setup.py`. ⚠ nh3-pve's AMT took every setting but does NOT dial out: it is on a - STATIC IP, and Intel's CIRA guidance says static IP does not work (environment detection keys on DHCP - option 15). Open decision: move it to DHCP (the UDM reservation already pins 10.100.250.61). + side is `scripts/amt-cira-setup.py`. **nh3-pve's AMT phones home since 0859** after moving it from static IP + to DHCP (Intel: CIRA does not work on a static IP; the FortiGate sniffer had shown zero attempts before). + The CIRA entry is `nh3-pve-amt` (the old LAN entry was removed). Two things it needed: the AMT + credentials set on the device (`changedevice` intelamt user/pass) and `intelamt.tls` = 1. Then a + MeshCentral restart re-ran its AMT manager, which logged in at once (16.1.25, power on). ⚠ MeshCentral + 1.2.0 `amtmanager.js` picks TLS-vs-not over CIRA with `boundPorts.indexOf('16992')` used as a boolean + (−1 is truthy), so a TLS-only AMT can be tried without TLS. Set `tls` explicitly as above. - Before 2026-10-02: `"WANonly": true` (TacticalRMM's install default). In that mode MeshCentral SILENTLY DROPS "Add Intel AMT computer": `meshuser.js` line 2682, `if (args.wanonly == true) return;`. No error, no event. LAN-mode AMT needs `WANonly` false (hybrid) + a service restart; CIRA works in WAN mode. TacticalRMM's