vh
b92097688c
esh-pve hard-froze at 03:34 on 2026-08-19 and stayed frozen ~4.5 hours until a manual power cycle. No panic, no OOM, no MCE — the journal stops mid-operation. The whole ESH site lost DNS with it, because esh-userland (VLAN 10, the PVC SSID and wired userland LAN) was handed exactly one resolver: 10.0.50.45, AdGuard on esh-docker-vm, on a different VLAN, with no secondary. Internet and routing were healthy throughout. Two fixes. 1. DNS: 10.0.10.1 (the gateway, verified resolving) added as secondary on esh-userland via the UDM Classic API. Note this is degradation cover, not clean failover — clients that query resolvers in parallel will bypass AdGuard for a share of lookups. 2. Watchdog: softdog -> iTCO_wdt under systemd (RuntimeWatchdogSec=60), watchdog-mux masked. The box looked watchdog-protected and was not: a software watchdog cannot fire when the kernel it lives in is wedged, and watchdog-mux only pets the device while an HA client is connected, which never happens on a cluster with no HA resources. Firmware does not block the TCO timer here, checked before committing to it. Also pins VM 102 off (onboot: 0). It starts with full GPU passthrough and vfio-pci enabling that device is the last thing the kernel logged, 39 minutes before the freeze. The other suspect is the kernel itself: the host ran 4.5 months on 6.8.12-16, took 6.8.12-42 in an apt batch on 08-18, and died 20 hours into the first boot on it. 6.8.12-16 is still installed as the rollback. The playbook is idempotent — a second run skips all six steps and passes all six verifies. The watchdog is confirmed armed (identity=iTCO_wdt, state=active, held by PID 1) but has NOT been observed firing; proving that needs a deliberate wedge. Memory also corrects two wrong mid-incident calls: the mgmt VLAN is routed over the site tunnel and is not firewalled off — both symptoms were the dead host generating ICMP unreachables.
esh-pve
Proxmox VE hypervisor for the ESH home lab (pve.esteban.net). Non-PFI scope.
Network
- LAN IP: 10.0.250.35
- FQDN:
pve.esteban.net - Web UI:
https://10.0.250.35:8006 - SSH:
root@esh-pve(key auth)
Hardware
- CPU: 13th Gen Intel Core i9-13900H
- RAM: 62.5 GB
- Kernel:
6.8.12-16-pve(Proxmox 8.x) - Storage: local
pve-rootat 94G (21% used) + two large NFS mounts from10.0.50.50:/mnt/pve/esh-nas— 92 TB available on/mnt/pvestore/mnt/pve/tank-vmbu— 93 TB, VM backup target (~1.4 TB used)
What it runs
esh-docker-vm (10.0.50.45) is a VM here. Other home-lab VMs are not yet catalogued in this workspace — run qm list on the host for the live inventory.
VM 102 esh-vm-workstation is pinned off (onboot: 0, stopped) as of
2026-08-19. It is on-demand and there has been no demand, and it is the prime
suspect in the freeze below — it starts with full GPU passthrough
(hostpci0: 0000:01:00,pcie=1,x-vga=1) and that was the last thing the kernel
logged before the host died. Starting it is the moment of risk.
Watchdog — hardware, not software
This host runs the PCH hardware watchdog (iTCO_wdt, 60 s, owned by
systemd via RuntimeWatchdogSec). softdog is blacklisted and Proxmox's
watchdog-mux is masked.
That is deliberate. On 2026-08-19 esh-pve hard-froze at 03:34 and stayed frozen
for ~4.5 hours until someone power-cycled it by hand — taking the only DNS
resolver the esh-userland VLAN is handed down with it. It was running
softdog at the time, which cannot rescue a hard freeze because the frozen
kernel is what would have to fire the timer; and watchdog-mux held the device
without ever arming it, because it only pets while an HA client is connected
and this cluster has no HA resources.
Applied and re-runnable via
playbooks/esh-pve-hardware-watchdog.yaml.
⚠️ Reverse this before configuring Proxmox HA on esh-pve — HA fencing needs
watchdog-mux to own /dev/watchdog. Not a near-term concern: this is a
two-node cluster with no qdevice, so losing one node already costs quorum.
⚠️ The watchdog is armed but has not been observed firing. Confirming it means deliberately wedging the host.
vPro / AMT — not currently usable
The board is vPro-capable, but AMT needs the chipset-integrated Intel PHY (one of the two i226 RJ45 ports) and this host reaches the network only over SFP+ (Intel X710 → port 27 on the Garage switch), presenting a single MAC. AMT cannot ride a discrete/SFP+ NIC. Cable an onboard RJ45 and provision AMT in MEBx to get out-of-band power control; until then, recovery is the hardware watchdog above or a physical trip.
Refresh state
scripts/refresh-server-info.sh esh-pve
Same Proxmox-inspect caveat as pfi-pve.
Placement rule
Hypervisor for ESH home-lab VMs. Not part of the PFI colo topology.
⚠ Cluster membership and the dark-tile gotcha
This node (pve) is half of the 2-node esh-pve-cluster with esh-nas-pve
(esh-pve-nas, 10.0.50.55). Expected votes: 2, Quorum: 2, no qdevice — so
rebooting either node makes the survivor's /etc/pve read-only until the
partner is back. Guests keep running; config changes do not. Plan reboots of
either node accordingly (pvecm expected 1 on the survivor, or accept the
read-only window).
⚠ A dark/greyed node tile usually means pvestatd, not a dead node.
pvestatd SEGV'd here on 2026-05-28 and stayed dead for 82 days — the node
was quorate and healthy the entire time, with pve-cluster, corosync,
pvedaemon, pveproxy and both HA services active and all three guests running.
It is only the reporting daemon, so its death is invisible except that the UI
has nothing to render. It has SEGV'd four times (2025-08-18, 2025-09-04,
2025-09-15, 2026-05-28) — treat a recurrence as expected, not novel.
systemctl reset-failed pvestatd && systemctl restart pvestatd
Restarted 2026-08-18. Worth a watchdog: nothing alerts when it dies, and the only symptom is a cosmetic one nobody looks at for months.