From 719e3fa3248551ae5db2feebf3992746a4ee00bc Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Thu, 24 Sep 2026 16:24:36 -0700 Subject: [PATCH] docs(esh-pve): VM 102 retired; record the hung D-state kvm from the 09-10 vzdump and tonight's reboot checklist --- servers/esh-pve/README.md | 32 +++++++++++++++++++++++++++----- 1 file changed, 27 insertions(+), 5 deletions(-) diff --git a/servers/esh-pve/README.md b/servers/esh-pve/README.md index a02fce1..08d7407 100644 --- a/servers/esh-pve/README.md +++ b/servers/esh-pve/README.md @@ -22,11 +22,33 @@ Proxmox VE hypervisor for the ESH home lab (`pve.esteban.net`). Non-PFI scope. `esh-docker-vm` (`10.0.50.45`) is a VM here. Other home-lab VMs are not yet catalogued in this workspace — run `qm list` on the host for the live inventory. -VM **102 `esh-vm-workstation` is pinned off** (`onboot: 0`, stopped) as of -2026-08-19. It is on-demand and there has been no demand, and it is the prime -suspect in the freeze below — it starts with full GPU passthrough -(`hostpci0: 0000:01:00,pcie=1,x-vga=1`) and that was the last thing the kernel -logged before the host died. Starting it is the moment of risk. +VM **102 `esh-vm-workstation` is RETIRED (2026-09-24, Prime: "not the machine I +want running a virtual Windows machine").** `qm destroy 102 --purge` removed its +config and EFI disk. Its final backup, `pbs-ana:backup/vm/102/2026-09-10T10:32:20Z` +(verification ok), is **protected** against prune, so the retirement is +restorable. It had full GPU passthrough (`hostpci0: 0000:01:00,pcie=1,x-vga=1`) +and was the prime suspect in the 2026-08-19 freeze below. + +⚠ **Found while retiring it: a hung VM-102 process since 2026-09-10 0332.** That +night's vzdump started the stopped VM with the T400 passed through to read its +disks, and the `kvm -id 102` process never exited. It is in **D state** (stuck in +`blkdev` teardown) together with its `swtpm` and a udev worker, so it cannot be +killed. It holds **16 GB of locked RAM**, the T400, and the two LVs `qm destroy` +could not remove (`pve/vm-102-disk-1` 256G, `pve/vm-102-disk-2` TPM). `qm status +102` said `stopped` the whole time. **Only a host reboot clears it.** It is the +same class of fault as the August freeze: VFIO plus this T400. + +**Reboot planned for the evening of 2026-09-24 with Prime on site.** Checklist: +1. Cluster: rebooting makes esh-pve-nas's `/etc/pve` read-only until this node is + back (guests keep running). Accept that, or run `pvecm expected 1` there first. +2. After boot, `lvremove pve/vm-102-disk-1 pve/vm-102-disk-2`, then confirm + ~16 GB RAM is freed and nothing holds `01:00.0`. The T400 stays bound to + vfio-pci, unassigned. +3. esh-docker-vm: watch the known NFS boot race and the traefik restart and + crowdsec fail-closed recovery (auto-memory incident notes). DNS for + esh-userland comes back with this host. +4. If Prime cables an onboard RJ45 (the **I226-LM** at `59:00.0` is the AMT port), + provision AMT in MEBx during the same window. See the vPro section. ## Watchdog — hardware, not software