From b92097688c2bd58719d2853f40ed4592380216d1 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 19 Aug 2026 09:05:52 -0700 Subject: [PATCH] fix(esh-pve): hardware watchdog, and close the single-resolver DNS SPOF MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit esh-pve hard-froze at 03:34 on 2026-08-19 and stayed frozen ~4.5 hours until a manual power cycle. No panic, no OOM, no MCE — the journal stops mid-operation. The whole ESH site lost DNS with it, because esh-userland (VLAN 10, the PVC SSID and wired userland LAN) was handed exactly one resolver: 10.0.50.45, AdGuard on esh-docker-vm, on a different VLAN, with no secondary. Internet and routing were healthy throughout. Two fixes. 1. DNS: 10.0.10.1 (the gateway, verified resolving) added as secondary on esh-userland via the UDM Classic API. Note this is degradation cover, not clean failover — clients that query resolvers in parallel will bypass AdGuard for a share of lookups. 2. Watchdog: softdog -> iTCO_wdt under systemd (RuntimeWatchdogSec=60), watchdog-mux masked. The box looked watchdog-protected and was not: a software watchdog cannot fire when the kernel it lives in is wedged, and watchdog-mux only pets the device while an HA client is connected, which never happens on a cluster with no HA resources. Firmware does not block the TCO timer here, checked before committing to it. Also pins VM 102 off (onboot: 0). It starts with full GPU passthrough and vfio-pci enabling that device is the last thing the kernel logged, 39 minutes before the freeze. The other suspect is the kernel itself: the host ran 4.5 months on 6.8.12-16, took 6.8.12-42 in an apt batch on 08-18, and died 20 hours into the first boot on it. 6.8.12-16 is still installed as the rollback. The playbook is idempotent — a second run skips all six steps and passes all six verifies. The watchdog is confirmed armed (identity=iTCO_wdt, state=active, held by PID 1) but has NOT been observed firing; proving that needs a deliberate wedge. Memory also corrects two wrong mid-incident calls: the mgmt VLAN is routed over the site tunnel and is not firewalled off — both symptoms were the dead host generating ICMP unreachables. --- .../2026-08-19-esh-pve-freeze-dns-spof.md | 108 ++++++++++++++++ persistent-memory.md | 4 + playbooks/esh-pve-hardware-watchdog.yaml | 120 ++++++++++++++++++ servers/esh-pve/README.md | 39 ++++++ 4 files changed, 271 insertions(+) create mode 100644 persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md create mode 100644 playbooks/esh-pve-hardware-watchdog.yaml diff --git a/persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md b/persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md new file mode 100644 index 0000000..40b74f8 --- /dev/null +++ b/persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md @@ -0,0 +1,108 @@ +# `[2026-08-19]` esh-pve hard-froze for 4.5h — and took the whole house's DNS with it + +Reported by the operator as "routing or DNS issues on the PVC wifi." It was +neither: the internet was healthy the entire time (gateway reporting 3 ms and +209/26 Mbps; 1.1.1.1 and 8.8.8.8 answering at ~3 ms from inside ESH with zero +loss). **The house had no name resolution because one VM was down.** + +## The SPOF: one resolver, cross-VLAN, no fallback + +`esh-userland` (VLAN 10, `10.0.10.0/24` — the `PVC` SSID *and* the wired +userland LAN) handed out **exactly one DNS server, `10.0.50.45`** — AdGuard, on +`esh-docker-vm`, on the **server** VLAN. No secondary. That VM dies, every +client on the VLAN loses DNS, and it presents as "the wifi is broken." + +It was the only network in the house exposed this way. `Default`, `esh-mgmt`, +`esh-server` and `esh-cameras` run DNS on auto (the gateway hands itself out); +`esh-iot` and `ESH-WG` point at 1.1.1.1 + 8.8.8.8. + +**Fixed** (operator-approved): `esh-userland` now hands out `10.0.50.45` +primary, **`10.0.10.1` (the gateway) secondary** — the UDM's own resolver, +verified answering. Applied via the Classic API, +`PUT /proxy/network/api/s/default/rest/networkconf/687985eae5d15b673cef1a73` +with the full object (GET → modify one field → PUT), `rc: ok`. **This was also +the first confirmed WRITE on the ESH UDM key** — previously only the NH3 key +was write-tested. See [[reference_unifi_udm_integration_api_keys]]. + +⚠️ **A secondary is not clean failover.** macOS/iOS query resolvers in +parallel, so once AdGuard is back a real share of lookups go to the gateway and +**skip ad-blocking**. This converts a total outage into degraded-but-working. +The actual fix for blocking integrity is a second AdGuard instance NOT on +esh-pve. + +## Root cause: hard freeze, no diagnostics, two suspects + +`esh-pve` (Minisforum MS-01, i9-13900H, `productname: YajuuSenpai`) froze at +**03:34:39**. The journal stops mid-operation — **no panic, no OOM, no MCE, no +thermal event**. Powered on with its 10G link up, but not answering ARP. + +Two changes landed the day before, and they are not exclusive: + +1. **New kernel.** A large `apt` batch on **2026-08-18 07:00:21** installed + `proxmox-kernel-6.8.12-42-pve`; clean reboot at 07:08:44. Before that the + box had **4.5 months of uptime** (Mar 30 → Aug 18) on `6.8.12-16`. First + boot on the new kernel lasted **20 hours**. +2. **GPU passthrough.** The last kernel messages of the dead boot are + `vfio-pci 0000:01:00.0/.1: enabling device` at **02:55:17** — VM 102 + `esh-vm-workstation` starting with `hostpci0: 0000:01:00,pcie=1,x-vga=1`, + **39 minutes before the freeze**. + +A vfio/i915 regression in the newer kernel would produce exactly this +signature. `6.8.12-16` is still installed and is the held-in-reserve rollback. + +**VM 102 is now pinned off** (`qm set 102 --onboot 0`, stopped) per the +operator — it is on-demand and there has been no demand. That removes the +suspect without a kernel rollback. + +## Why nobody could recover it remotely — and the fix + +Nothing on the box could reboot it: + +- **`softdog` was the loaded watchdog.** A *software* watchdog cannot rescue a + hard kernel freeze: the frozen kernel is the thing that would have to fire + its timer. This is the trap — the machine *looked* watchdog-protected. +- **Proxmox's `watchdog-mux` held `/dev/watchdog` but never armed it.** It only + pets the device while an HA client is connected, and this cluster has no HA + resources. +- **vPro/AMT was unusable.** The MS-01 reaches the network only via **SFP+** + (Intel X710, port 27 on the Garage switch) and presents exactly one MAC. + **AMT cannot ride a discrete/SFP+ NIC** — it needs the chipset-integrated + Intel PHY, i.e. one of the two i226 RJ45 ports, and both are unplugged. + Cabling one and provisioning AMT in MEBx remains the open item for *control*; + the watchdog below is the fix for *recovery*. + +**Fixed:** `playbooks/esh-pve-hardware-watchdog.yaml` — systemd now owns the +PCH hardware watchdog (`iTCO_wdt`, `RuntimeWatchdogSec=60`), `softdog` is +blacklisted and unloaded, `watchdog-mux` is masked. Verified live: +`watchdog0: identity=iTCO_wdt state=active timeout=60s`, held by PID 1, +journal `Using hardware watchdog 'iTCO_wdt', version 6`. Playbook re-run proves +idempotency (6 skipped / 6 verify OK). + +Firmware does **not** block the TCO timer here — checked for the +`unable to reset NO_REBOOT flag` line before committing to the approach; the +board reports `Found a Intel PCH TCO device (Version=6, TCOBASE=0x0400)`. + +⚠️ **Masking `watchdog-mux` trades away HA fencing.** If Proxmox HA is ever +configured on esh-pve this must be reverted. Not a near-term concern: +`esh-pve-cluster` is **two nodes with no qdevice**, so a single node loss +already costs quorum and the survivor would fence itself — HA here would reduce +availability, not raise it. + +⚠️ **The watchdog is configured and armed, but has NOT been proven to fire.** +Proving it means deliberately wedging the host. Untested-but-armed is still +strictly better than softdog; treat a real firing as unconfirmed until tested. + +## Diagnostic corrections worth keeping + +- **"No route to host" was the dead host, not a routing gap.** Two claims made + mid-incident were wrong: that the mgmt VLAN (`10.0.250.0/24`) is not routed + over the NH3↔ESH tunnel, and that a firewall isolates it from the server + VLAN. Both were artifacts of esh-pve being dead. With it up, `root@esh-pve` + SSHes fine from nh3-dev at 7.5 ms, and `10.0.250.1` answers from + `esh-pve-nas` in 0.078 ms. **Control-test against a *different* host on the + target subnet before concluding "the subnet is unreachable."** +- **UDM `uptime` on a client record is association time, not host uptime.** It + read 2.2 days while the host had been up 20 hours. Use + `journalctl --list-boots` on the host for real boot history. +- **`rest/user` `last_seen` is not maintained** (it read ~203 days for hosts + that are demonstrably online). `stat/sta` is the live view. diff --git a/persistent-memory.md b/persistent-memory.md index 1d6098f..b785448 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -109,6 +109,8 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight +- **🔴→🟢 ESH OUTAGE 2026-08-19 — esh-pve hard-froze 03:34, ~4.5h, whole house lost DNS.** Presented as "wifi/routing issues"; internet was healthy throughout. Cause: `esh-userland` (VLAN 10, the `PVC` SSID) handed out **one** resolver, `10.0.50.45` (AdGuard on `esh-docker-vm`) — cross-VLAN, no secondary — and its hypervisor died. **Recovered by hand power-cycle; all VMs, cluster quorum and DNS restored.** Two fixes landed: gateway `10.0.10.1` added as secondary DNS on esh-userland (operator-approved, first confirmed WRITE on the ESH UDM key), and **`softdog` → `iTCO_wdt` hardware watchdog owned by systemd** (`playbooks/esh-pve-hardware-watchdog.yaml`, idempotent, verified armed) so a repeat self-recovers in 60s. **VM 102 pinned off** (`onboot: 0`) as the passthrough suspect. **⏳ OPEN:** (a) the watchdog is armed but **has not been proven to fire** — needs a deliberate wedge to confirm; (b) AMT/vPro still unusable until an onboard **RJ45** is cabled (the MS-01 is SFP+-only on the network and AMT cannot ride it); (c) kernel `6.8.12-16` rollback held in reserve if it freezes again. → `persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md` + _As of 2026-08-19 (late) — a long infra session, all of it landed and committed: **fleet `*.internal` DNS is live** (git-sourced, 42 names, three resolvers incl. a NEW colo one), **waterland studio containerised** on irv-ml1, **Homepage cleaned up and themed** with Australis Skyfall + an Arbo-generated background, and **four unmanaged stacks adopted** into `stacks/` (two of which turned out to be quietly broken — SearXNG's healthcheck, SeaFile down 3 months). ⚠ 17 commits unpushed. ~~⚠ 2 unread althing messages from waterland-dev left deliberately undrained~~ — **drained 2026-08-19 post-`/clear`; thread closed, see below.**_ - **🟢 FLEET `.internal` DNS — LIVE 2026-08-19.** `..internal`, sites `ana`/`esh`/`nh3`. `dns/internal.yaml` is the source of truth; `scripts/dns-sync.py` reconciles the three AdGuard resolvers (diff → prompt → apply, idempotent). 42 names resolving from all three sites. **Colo got its first resolver ever** (`stacks/adguard-ana/`, API on **8053** not 8080, no blocklists by design) — before this, ana-docker resolved straight against `1.1.1.1`. Auth = a dedicated `infra-ops` AdGuard user, password vaulted `nh3-dev/adguard-infra-ops-password`. **⏳ TWO OPEN, both operator's to schedule:** (a) colo hosts still point at `1.1.1.1` so they do not yet *use* the new resolver — repointing a site's DNS is a separate change; (b) the static-v6 convention (each server at its site's `/64` with low bits echoing the v4 octet, `esh-docker-vm` → `…::45`) is **proposed, not ruled on**. The `v6:` column is empty and correct — no fleet host has a global v6 address yet. → `persistent-memory.d/2026-08-19-fleet-internal-dns.md` @@ -164,6 +166,8 @@ _As of 2026-08-19 (late) — a long infra session, all of it landed and committe ## Recent decisions +- `[2026-08-19]` **A *software* watchdog is not watchdog protection — esh-pve froze for 4.5h holding one.** softdog cannot fire when the kernel it runs in is wedged, and Proxmox's `watchdog-mux` never arms without HA resources, so the box *looked* protected and wasn't. Moved to the PCH `iTCO_wdt` under systemd. Also: a single cross-VLAN DNS entry with no secondary turns any VM outage into a whole-site outage. → `persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md` + - `[2026-08-19]` **Fleet `.internal` DNS built and live — git-sourced, agent-managed, three resolvers.** Zone-scoped authority (ESH's hand-made `esteban.net` rewrites survive); the colo had no resolver at all; v6 column empty on purpose because SLAAC addresses rotate. → `persistent-memory.d/2026-08-19-fleet-internal-dns.md` - `[2026-08-19]` **waterland studio containerised on irv-ml1 — three landmines, all measured.** cupy needs CUDA *headers* the host had by accident; `uv run` re-syncs and prunes cupy at RUNTIME; the A6000 is container-index 0, not the host's 1. → `persistent-memory.d/2026-08-19-waterland-studio-containerised.md` diff --git a/playbooks/esh-pve-hardware-watchdog.yaml b/playbooks/esh-pve-hardware-watchdog.yaml new file mode 100644 index 0000000..0042ea0 --- /dev/null +++ b/playbooks/esh-pve-hardware-watchdog.yaml @@ -0,0 +1,120 @@ +# esh-pve — move from the software watchdog to the PCH hardware watchdog. +# +# WHY: esh-pve hard-froze at 03:34 on 2026-08-19 (no panic, no OOM, no MCE — +# the journal simply stops mid-line) and stayed frozen for ~4.5 hours until it +# was power-cycled by hand. Everything on it went with it, including the only +# DNS resolver the esh-userland VLAN is handed, so the whole house lost name +# resolution. +# +# Nothing on the box could have recovered it: +# - `softdog` was the loaded watchdog. A SOFTWARE watchdog cannot rescue a +# hard kernel freeze, because the frozen kernel is the thing that would +# have to fire its timer. +# - Proxmox's `watchdog-mux` held /dev/watchdog but never armed it: it only +# pets the device while an HA client is connected, and this cluster has no +# HA resources configured (`ha-manager status` reports quorum only). +# +# The board's PCH TCO timer is present and NOT blocked by firmware — verified +# before writing this: +# iTCO_wdt: Found a Intel PCH TCO device (Version=6, TCOBASE=0x0400) +# iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0) +# (no "unable to reset NO_REBOOT flag" line, which is the BIOS-blocked case). +# +# APPROACH: systemd owns the hardware watchdog directly. Setting +# WATCHDOG_MODULE=iTCO_wdt in /etc/default/pve-ha-manager would point +# watchdog-mux at the right device but would still never arm it without HA, so +# it does not solve this. systemd's RuntimeWatchdogSec pets unconditionally, +# which is what "reboot me if I wedge" actually requires. +# +# ⚠️ CONSEQUENCE — READ BEFORE ENABLING PROXMOX HA ON THIS CLUSTER. +# This masks `watchdog-mux`. If HA is ever configured on esh-pve, watchdog-mux +# must own /dev/watchdog again and this must be reverted, or HA fencing will +# not work. That is not a near-term concern: esh-pve-cluster is TWO nodes with +# no qdevice, so a single node loss already costs quorum and the survivor would +# fence itself. HA here would make availability worse, not better. +# +# Revert: unmask + enable watchdog-mux, delete the three dropped files, +# `systemctl daemon-reexec`, reboot. +# +# Run: scripts/elway esh-pve --playbook playbooks/esh-pve-hardware-watchdog.yaml + +vars: + # 60s: long enough that a busy-but-healthy host is never reset, short enough + # that a freeze costs a minute rather than half a working day. systemd pets + # at half this interval. PID 1 does not block on filesystem I/O, so the known + # NFS-wedge history on this host does not put it at risk of a false trip. + runtime_watchdog_sec: 60 + +steps: + - name: Load iTCO_wdt at every boot + shell: | + printf '# PCH hardware watchdog — see playbooks/esh-pve-hardware-watchdog.yaml\niTCO_wdt\n' \ + > /etc/modules-load.d/itco-watchdog.conf + when: "! grep -qx 'iTCO_wdt' /etc/modules-load.d/itco-watchdog.conf 2>/dev/null" + + - name: Stop softdog being auto-loaded so iTCO_wdt claims watchdog0 + shell: | + printf '# softdog cannot rescue a hard freeze; iTCO_wdt can.\n# See playbooks/esh-pve-hardware-watchdog.yaml\nblacklist softdog\n' \ + > /etc/modprobe.d/blacklist-softdog.conf + when: "! grep -qx 'blacklist softdog' /etc/modprobe.d/blacklist-softdog.conf 2>/dev/null" + + - name: Mask watchdog-mux (idle without HA, and it holds the device) + shell: systemctl disable --now watchdog-mux.service && systemctl mask watchdog-mux.service + when: "[ \"$(systemctl is-enabled watchdog-mux.service 2>/dev/null)\" != masked ]" + + - name: Hand the watchdog to systemd + shell: | + mkdir -p /etc/systemd/system.conf.d + cat > /etc/systemd/system.conf.d/watchdog.conf <<'EOF' + # Hardware watchdog (iTCO_wdt). See playbooks/esh-pve-hardware-watchdog.yaml + # for why systemd owns this rather than Proxmox's watchdog-mux. + [Manager] + RuntimeWatchdogSec={{ runtime_watchdog_sec }} + RebootWatchdogSec=10min + EOF + when: "! grep -q 'RuntimeWatchdogSec={{ runtime_watchdog_sec }}' /etc/systemd/system.conf.d/watchdog.conf 2>/dev/null" + + # Renumber live so the change takes effect without waiting for a reboot: + # softdog currently holds watchdog0, so systemd would otherwise arm the + # software watchdog — the exact device that failed us. + - name: Make iTCO_wdt the active watchdog0 now + shell: | + rmmod softdog 2>/dev/null || true + rmmod iTCO_wdt 2>/dev/null || true + modprobe iTCO_wdt + # Gated, not unconditional: once systemd holds /dev/watchdog0 the rmmod + # would fail anyway, and re-running this on an already-correct host would + # otherwise churn the device for no reason. Only renumber when watchdog0 + # is NOT already iTCO_wdt. + when: "! grep -qx 'iTCO_wdt' /sys/class/watchdog/watchdog0/identity 2>/dev/null" + + - name: Re-exec systemd so RuntimeWatchdogSec takes effect + # daemon-reload does NOT apply [Manager] settings; a re-exec is required. + # Skipped once systemd already reports it owns the hardware watchdog. + shell: systemctl daemon-reexec + when: "! journalctl -b --no-pager | grep -q 'Using hardware watchdog .iTCO_wdt.'" + +verify: + - name: iTCO_wdt is the kernel's watchdog0 + shell: grep -qx 'iTCO_wdt' /sys/class/watchdog/watchdog0/identity + changed_when: "false" + + - name: The watchdog is ARMED, not merely present + shell: grep -qx 'active' /sys/class/watchdog/watchdog0/state + changed_when: "false" + + - name: systemd reports it owns a hardware watchdog + shell: journalctl -b --no-pager | grep -q 'Using hardware watchdog .iTCO_wdt.' + changed_when: "false" + + - name: softdog is not loaded + shell: "! lsmod | grep -qE '^softdog'" + changed_when: "false" + + - name: watchdog-mux is masked + shell: "[ \"$(systemctl is-enabled watchdog-mux.service 2>/dev/null)\" = masked ]" + changed_when: "false" + + - name: Config survives a reboot + shell: grep -qx 'iTCO_wdt' /etc/modules-load.d/itco-watchdog.conf && grep -q RuntimeWatchdogSec /etc/systemd/system.conf.d/watchdog.conf + changed_when: "false" diff --git a/servers/esh-pve/README.md b/servers/esh-pve/README.md index fdb1fdc..a02fce1 100644 --- a/servers/esh-pve/README.md +++ b/servers/esh-pve/README.md @@ -22,6 +22,45 @@ Proxmox VE hypervisor for the ESH home lab (`pve.esteban.net`). Non-PFI scope. `esh-docker-vm` (`10.0.50.45`) is a VM here. Other home-lab VMs are not yet catalogued in this workspace — run `qm list` on the host for the live inventory. +VM **102 `esh-vm-workstation` is pinned off** (`onboot: 0`, stopped) as of +2026-08-19. It is on-demand and there has been no demand, and it is the prime +suspect in the freeze below — it starts with full GPU passthrough +(`hostpci0: 0000:01:00,pcie=1,x-vga=1`) and that was the last thing the kernel +logged before the host died. Starting it is the moment of risk. + +## Watchdog — hardware, not software + +This host runs the **PCH hardware watchdog** (`iTCO_wdt`, 60 s, owned by +systemd via `RuntimeWatchdogSec`). `softdog` is blacklisted and Proxmox's +`watchdog-mux` is **masked**. + +That is deliberate. On 2026-08-19 esh-pve hard-froze at 03:34 and stayed frozen +for ~4.5 hours until someone power-cycled it by hand — taking the only DNS +resolver the `esh-userland` VLAN is handed down with it. It was running +`softdog` at the time, which cannot rescue a hard freeze because the frozen +kernel is what would have to fire the timer; and `watchdog-mux` held the device +without ever arming it, because it only pets while an HA client is connected +and this cluster has no HA resources. + +Applied and re-runnable via +[`playbooks/esh-pve-hardware-watchdog.yaml`](../../playbooks/esh-pve-hardware-watchdog.yaml). + +⚠️ **Reverse this before configuring Proxmox HA on esh-pve** — HA fencing needs +`watchdog-mux` to own `/dev/watchdog`. Not a near-term concern: this is a +two-node cluster with no qdevice, so losing one node already costs quorum. + +⚠️ The watchdog is armed but **has not been observed firing**. Confirming it +means deliberately wedging the host. + +## vPro / AMT — not currently usable + +The board is vPro-capable, but AMT needs the chipset-integrated Intel PHY (one +of the two i226 **RJ45** ports) and this host reaches the network only over +**SFP+** (Intel X710 → port 27 on the Garage switch), presenting a single MAC. +AMT cannot ride a discrete/SFP+ NIC. Cable an onboard RJ45 and provision AMT in +MEBx to get out-of-band power control; until then, recovery is the hardware +watchdog above or a physical trip. + ## Refresh state ```bash