esh-pve

Proxmox VE hypervisor for the ESH home lab (pve.esteban.net). Non-PFI scope.

Network

LAN IP: 10.0.250.35
FQDN: pve.esteban.net
Web UI: https://10.0.250.35:8006
SSH: root@esh-pve (key auth)

Hardware

CPU: 13th Gen Intel Core i9-13900H
RAM: 62.5 GB
Kernel: 6.8.12-16-pve (Proxmox 8.x)
Storage: local pve-root at 94G (21% used) + two large NFS mounts from 10.0.50.50:
- /mnt/pve/esh-nas — 92 TB available on /mnt/pvestore
- /mnt/pve/tank-vmbu — 93 TB, VM backup target (~1.4 TB used)

What it runs

esh-docker-vm (10.0.50.45) is a VM here. Other home-lab VMs are not yet catalogued in this workspace — run qm list on the host for the live inventory.

VM 102 esh-vm-workstation is RETIRED (2026-09-24, Prime: "not the machine I want running a virtual Windows machine"). qm destroy 102 --purge removed its config and EFI disk. Its final backup, pbs-ana:backup/vm/102/2026-09-10T10:32:20Z (verification ok), is protected against prune, so the retirement is restorable. It had full GPU passthrough (hostpci0: 0000:01:00,pcie=1,x-vga=1) and was the prime suspect in the 2026-08-19 freeze below.

⚠ Found while retiring it: a hung VM-102 process since 2026-09-10 0332. That night's vzdump started the stopped VM with the T400 passed through to read its disks, and the kvm -id 102 process never exited. It is in D state (stuck in blkdev teardown) together with its swtpm and a udev worker, so it cannot be killed. It holds 16 GB of locked RAM, the T400, and the two LVs qm destroy could not remove (pve/vm-102-disk-1 256G, pve/vm-102-disk-2 TPM). qm status 102 said stopped the whole time. Only a host reboot clears it. It is the same class of fault as the August freeze: VFIO plus this T400.

Reboot planned for the evening of 2026-09-24 with Prime on site. Checklist:

Cluster: rebooting makes esh-pve-nas's /etc/pve read-only until this node is back (guests keep running). Accept that, or run pvecm expected 1 there first.
After boot, lvremove pve/vm-102-disk-1 pve/vm-102-disk-2, then confirm ~16 GB RAM is freed and nothing holds 01:00.0. The T400 stays bound to vfio-pci, unassigned.
esh-docker-vm: watch the known NFS boot race and the traefik restart and crowdsec fail-closed recovery (auto-memory incident notes). DNS for esh-userland comes back with this host.
If Prime cables an onboard RJ45 (the I226-LM at 59:00.0 is the AMT port), provision AMT in MEBx during the same window. See the vPro section.

Watchdog — hardware, not software

This host runs the PCH hardware watchdog (iTCO_wdt, 60 s, owned by systemd via RuntimeWatchdogSec). softdog is blacklisted and Proxmox's watchdog-mux is masked.

That is deliberate. On 2026-08-19 esh-pve hard-froze at 03:34 and stayed frozen for ~4.5 hours until someone power-cycled it by hand — taking the only DNS resolver the esh-userland VLAN is handed down with it. It was running softdog at the time, which cannot rescue a hard freeze because the frozen kernel is what would have to fire the timer; and watchdog-mux held the device without ever arming it, because it only pets while an HA client is connected and this cluster has no HA resources.

Applied and re-runnable via playbooks/esh-pve-hardware-watchdog.yaml.

⚠️ Reverse this before configuring Proxmox HA on esh-pve — HA fencing needs watchdog-mux to own /dev/watchdog. Not a near-term concern: this is a two-node cluster with no qdevice, so losing one node already costs quorum.

⚠️ The watchdog is armed but has not been observed firing. Confirming it means deliberately wedging the host.

vPro / AMT — not currently usable

The board is vPro-capable, but AMT needs the chipset-integrated Intel PHY (one of the two i226 RJ45 ports) and this host reaches the network only over SFP+ (Intel X710 → port 27 on the Garage switch), presenting a single MAC. AMT cannot ride a discrete/SFP+ NIC. Cable an onboard RJ45 and provision AMT in MEBx to get out-of-band power control; until then, recovery is the hardware watchdog above or a physical trip.

Refresh state

scripts/refresh-server-info.sh esh-pve

Same Proxmox-inspect caveat as pfi-pve.

Placement rule

Hypervisor for ESH home-lab VMs. Not part of the PFI colo topology.

⚠ Cluster membership and the dark-tile gotcha

This node (pve) is half of the 2-node esh-pve-cluster with esh-nas-pve (esh-pve-nas, 10.0.50.55). Expected votes: 2, Quorum: 2, no qdevice — so rebooting either node makes the survivor's /etc/pve read-only until the partner is back. Guests keep running; config changes do not. Plan reboots of either node accordingly (pvecm expected 1 on the survivor, or accept the read-only window).

⚠ A dark/greyed node tile usually means pvestatd, not a dead node. pvestatd SEGV'd here on 2026-05-28 and stayed dead for 82 days — the node was quorate and healthy the entire time, with pve-cluster, corosync, pvedaemon, pveproxy and both HA services active and all three guests running. It is only the reporting daemon, so its death is invisible except that the UI has nothing to render. It has SEGV'd four times (2025-08-18, 2025-09-04, 2025-09-15, 2026-05-28) — treat a recurrence as expected, not novel.

systemctl reset-failed pvestatd && systemctl restart pvestatd

Restarted 2026-08-18. Worth a watchdog: nothing alerts when it dies, and the only symptom is a cosmetic one nobody looks at for months.

5.5 KiB Raw Blame History