docs(esh-pve): VM 102 retired; record the hung D-state kvm from the 09-10 vzdump and tonight's reboot checklist

servers/esh-pve/README.md

@@ -22,11 +22,33 @@ Proxmox VE hypervisor for the ESH home lab (`pve.esteban.net`). Non-PFI scope.
 
 `esh-docker-vm` (`10.0.50.45`) is a VM here. Other home-lab VMs are not yet catalogued in this workspace — run `qm list` on the host for the live inventory.
 
-VM **102 `esh-vm-workstation` is pinned off** (`onboot: 0`, stopped) as of
+VM **102 `esh-vm-workstation` is RETIRED (2026-09-24, Prime: "not the machine I
-2026-08-19. It is on-demand and there has been no demand, and it is the prime
+want running a virtual Windows machine").** `qm destroy 102 --purge` removed its
-suspect in the freeze below — it starts with full GPU passthrough
+config and EFI disk. Its final backup, `pbs-ana:backup/vm/102/2026-09-10T10:32:20Z`
-(`hostpci0: 0000:01:00,pcie=1,x-vga=1`) and that was the last thing the kernel
+(verification ok), is **protected** against prune, so the retirement is
-logged before the host died. Starting it is the moment of risk.
+restorable. It had full GPU passthrough (`hostpci0: 0000:01:00,pcie=1,x-vga=1`)
+and was the prime suspect in the 2026-08-19 freeze below.
+
+⚠ **Found while retiring it: a hung VM-102 process since 2026-09-10 0332.** That
+night's vzdump started the stopped VM with the T400 passed through to read its
+disks, and the `kvm -id 102` process never exited. It is in **D state** (stuck in
+`blkdev` teardown) together with its `swtpm` and a udev worker, so it cannot be
+killed. It holds **16 GB of locked RAM**, the T400, and the two LVs `qm destroy`
+could not remove (`pve/vm-102-disk-1` 256G, `pve/vm-102-disk-2` TPM). `qm status
+102` said `stopped` the whole time. **Only a host reboot clears it.** It is the
+same class of fault as the August freeze: VFIO plus this T400.
+
+**Reboot planned for the evening of 2026-09-24 with Prime on site.** Checklist:
+1. Cluster: rebooting makes esh-pve-nas's `/etc/pve` read-only until this node is
+   back (guests keep running). Accept that, or run `pvecm expected 1` there first.
+2. After boot, `lvremove pve/vm-102-disk-1 pve/vm-102-disk-2`, then confirm
+   ~16 GB RAM is freed and nothing holds `01:00.0`. The T400 stays bound to
+   vfio-pci, unassigned.
+3. esh-docker-vm: watch the known NFS boot race and the traefik restart and
+   crowdsec fail-closed recovery (auto-memory incident notes). DNS for
+   esh-userland comes back with this host.
+4. If Prime cables an onboard RJ45 (the **I226-LM** at `59:00.0` is the AMT port),
+   provision AMT in MEBx during the same window. See the vPro section.
 
 ## Watchdog — hardware, not software