esh-pve hard-froze at 03:34 on 2026-08-19 and stayed frozen ~4.5 hours
until a manual power cycle. No panic, no OOM, no MCE — the journal stops
mid-operation. The whole ESH site lost DNS with it, because esh-userland
(VLAN 10, the PVC SSID and wired userland LAN) was handed exactly one
resolver: 10.0.50.45, AdGuard on esh-docker-vm, on a different VLAN, with
no secondary. Internet and routing were healthy throughout.
Two fixes.
1. DNS: 10.0.10.1 (the gateway, verified resolving) added as secondary on
esh-userland via the UDM Classic API. Note this is degradation cover,
not clean failover — clients that query resolvers in parallel will
bypass AdGuard for a share of lookups.
2. Watchdog: softdog -> iTCO_wdt under systemd (RuntimeWatchdogSec=60),
watchdog-mux masked. The box looked watchdog-protected and was not: a
software watchdog cannot fire when the kernel it lives in is wedged,
and watchdog-mux only pets the device while an HA client is connected,
which never happens on a cluster with no HA resources. Firmware does
not block the TCO timer here, checked before committing to it.
Also pins VM 102 off (onboot: 0). It starts with full GPU passthrough and
vfio-pci enabling that device is the last thing the kernel logged, 39
minutes before the freeze. The other suspect is the kernel itself: the
host ran 4.5 months on 6.8.12-16, took 6.8.12-42 in an apt batch on
08-18, and died 20 hours into the first boot on it. 6.8.12-16 is still
installed as the rollback.
The playbook is idempotent — a second run skips all six steps and passes
all six verifies. The watchdog is confirmed armed (identity=iTCO_wdt,
state=active, held by PID 1) but has NOT been observed firing; proving
that needs a deliberate wedge.
Memory also corrects two wrong mid-incident calls: the mgmt VLAN is
routed over the site tunnel and is not firewalled off — both symptoms
were the dead host generating ICMP unreachables.
vh
b92097688c