fix(esh-pve): hardware watchdog, and close the single-resolver DNS SPOF

esh-pve hard-froze at 03:34 on 2026-08-19 and stayed frozen ~4.5 hours
until a manual power cycle. No panic, no OOM, no MCE — the journal stops
mid-operation. The whole ESH site lost DNS with it, because esh-userland
(VLAN 10, the PVC SSID and wired userland LAN) was handed exactly one
resolver: 10.0.50.45, AdGuard on esh-docker-vm, on a different VLAN, with
no secondary. Internet and routing were healthy throughout.

Two fixes.

1. DNS: 10.0.10.1 (the gateway, verified resolving) added as secondary on
   esh-userland via the UDM Classic API. Note this is degradation cover,
   not clean failover — clients that query resolvers in parallel will
   bypass AdGuard for a share of lookups.

2. Watchdog: softdog -> iTCO_wdt under systemd (RuntimeWatchdogSec=60),
   watchdog-mux masked. The box looked watchdog-protected and was not: a
   software watchdog cannot fire when the kernel it lives in is wedged,
   and watchdog-mux only pets the device while an HA client is connected,
   which never happens on a cluster with no HA resources. Firmware does
   not block the TCO timer here, checked before committing to it.

Also pins VM 102 off (onboot: 0). It starts with full GPU passthrough and
vfio-pci enabling that device is the last thing the kernel logged, 39
minutes before the freeze. The other suspect is the kernel itself: the
host ran 4.5 months on 6.8.12-16, took 6.8.12-42 in an apt batch on
08-18, and died 20 hours into the first boot on it. 6.8.12-16 is still
installed as the rollback.

The playbook is idempotent — a second run skips all six steps and passes
all six verifies. The watchdog is confirmed armed (identity=iTCO_wdt,
state=active, held by PID 1) but has NOT been observed firing; proving
that needs a deliberate wedge.

Memory also corrects two wrong mid-incident calls: the mgmt VLAN is
routed over the site tunnel and is not firewalled off — both symptoms
were the dead host generating ICMP unreachables.

servers/esh-pve/README.md

@@ -22,6 +22,45 @@ Proxmox VE hypervisor for the ESH home lab (`pve.esteban.net`). Non-PFI scope.
 
 `esh-docker-vm` (`10.0.50.45`) is a VM here. Other home-lab VMs are not yet catalogued in this workspace — run `qm list` on the host for the live inventory.
 
+VM **102 `esh-vm-workstation` is pinned off** (`onboot: 0`, stopped) as of
+2026-08-19. It is on-demand and there has been no demand, and it is the prime
+suspect in the freeze below — it starts with full GPU passthrough
+(`hostpci0: 0000:01:00,pcie=1,x-vga=1`) and that was the last thing the kernel
+logged before the host died. Starting it is the moment of risk.
+
+## Watchdog — hardware, not software
+
+This host runs the **PCH hardware watchdog** (`iTCO_wdt`, 60 s, owned by
+systemd via `RuntimeWatchdogSec`). `softdog` is blacklisted and Proxmox's
+`watchdog-mux` is **masked**.
+
+That is deliberate. On 2026-08-19 esh-pve hard-froze at 03:34 and stayed frozen
+for ~4.5 hours until someone power-cycled it by hand — taking the only DNS
+resolver the `esh-userland` VLAN is handed down with it. It was running
+`softdog` at the time, which cannot rescue a hard freeze because the frozen
+kernel is what would have to fire the timer; and `watchdog-mux` held the device
+without ever arming it, because it only pets while an HA client is connected
+and this cluster has no HA resources.
+
+Applied and re-runnable via
+[`playbooks/esh-pve-hardware-watchdog.yaml`](../../playbooks/esh-pve-hardware-watchdog.yaml).
+
+⚠️ **Reverse this before configuring Proxmox HA on esh-pve** — HA fencing needs
+`watchdog-mux` to own `/dev/watchdog`. Not a near-term concern: this is a
+two-node cluster with no qdevice, so losing one node already costs quorum.
+
+⚠️ The watchdog is armed but **has not been observed firing**. Confirming it
+means deliberately wedging the host.
+
+## vPro / AMT — not currently usable
+
+The board is vPro-capable, but AMT needs the chipset-integrated Intel PHY (one
+of the two i226 **RJ45** ports) and this host reaches the network only over
+**SFP+** (Intel X710 → port 27 on the Garage switch), presenting a single MAC.
+AMT cannot ride a discrete/SFP+ NIC. Cable an onboard RJ45 and provision AMT in
+MEBx to get out-of-band power control; until then, recovery is the hardware
+watchdog above or a physical trip.
+
 ## Refresh state
 
 ```bash