vh
b92097688c
esh-pve hard-froze at 03:34 on 2026-08-19 and stayed frozen ~4.5 hours until a manual power cycle. No panic, no OOM, no MCE — the journal stops mid-operation. The whole ESH site lost DNS with it, because esh-userland (VLAN 10, the PVC SSID and wired userland LAN) was handed exactly one resolver: 10.0.50.45, AdGuard on esh-docker-vm, on a different VLAN, with no secondary. Internet and routing were healthy throughout. Two fixes. 1. DNS: 10.0.10.1 (the gateway, verified resolving) added as secondary on esh-userland via the UDM Classic API. Note this is degradation cover, not clean failover — clients that query resolvers in parallel will bypass AdGuard for a share of lookups. 2. Watchdog: softdog -> iTCO_wdt under systemd (RuntimeWatchdogSec=60), watchdog-mux masked. The box looked watchdog-protected and was not: a software watchdog cannot fire when the kernel it lives in is wedged, and watchdog-mux only pets the device while an HA client is connected, which never happens on a cluster with no HA resources. Firmware does not block the TCO timer here, checked before committing to it. Also pins VM 102 off (onboot: 0). It starts with full GPU passthrough and vfio-pci enabling that device is the last thing the kernel logged, 39 minutes before the freeze. The other suspect is the kernel itself: the host ran 4.5 months on 6.8.12-16, took 6.8.12-42 in an apt batch on 08-18, and died 20 hours into the first boot on it. 6.8.12-16 is still installed as the rollback. The playbook is idempotent — a second run skips all six steps and passes all six verifies. The watchdog is confirmed armed (identity=iTCO_wdt, state=active, held by PID 1) but has NOT been observed firing; proving that needs a deliberate wedge. Memory also corrects two wrong mid-incident calls: the mgmt VLAN is routed over the site tunnel and is not firewalled off — both symptoms were the dead host generating ICMP unreachables.
97 lines
4.1 KiB
Markdown
97 lines
4.1 KiB
Markdown
# esh-pve
|
|
|
|
Proxmox VE hypervisor for the ESH home lab (`pve.esteban.net`). Non-PFI scope.
|
|
|
|
## Network
|
|
|
|
- **LAN IP:** 10.0.250.35
|
|
- **FQDN:** `pve.esteban.net`
|
|
- **Web UI:** `https://10.0.250.35:8006`
|
|
- **SSH:** `root@esh-pve` (key auth)
|
|
|
|
## Hardware
|
|
|
|
- **CPU:** 13th Gen Intel Core i9-13900H
|
|
- **RAM:** 62.5 GB
|
|
- **Kernel:** `6.8.12-16-pve` (Proxmox 8.x)
|
|
- **Storage:** local `pve-root` at 94G (21% used) + two large NFS mounts from `10.0.50.50`:
|
|
- `/mnt/pve/esh-nas` — 92 TB available on `/mnt/pvestore`
|
|
- `/mnt/pve/tank-vmbu` — 93 TB, VM backup target (~1.4 TB used)
|
|
|
|
## What it runs
|
|
|
|
`esh-docker-vm` (`10.0.50.45`) is a VM here. Other home-lab VMs are not yet catalogued in this workspace — run `qm list` on the host for the live inventory.
|
|
|
|
VM **102 `esh-vm-workstation` is pinned off** (`onboot: 0`, stopped) as of
|
|
2026-08-19. It is on-demand and there has been no demand, and it is the prime
|
|
suspect in the freeze below — it starts with full GPU passthrough
|
|
(`hostpci0: 0000:01:00,pcie=1,x-vga=1`) and that was the last thing the kernel
|
|
logged before the host died. Starting it is the moment of risk.
|
|
|
|
## Watchdog — hardware, not software
|
|
|
|
This host runs the **PCH hardware watchdog** (`iTCO_wdt`, 60 s, owned by
|
|
systemd via `RuntimeWatchdogSec`). `softdog` is blacklisted and Proxmox's
|
|
`watchdog-mux` is **masked**.
|
|
|
|
That is deliberate. On 2026-08-19 esh-pve hard-froze at 03:34 and stayed frozen
|
|
for ~4.5 hours until someone power-cycled it by hand — taking the only DNS
|
|
resolver the `esh-userland` VLAN is handed down with it. It was running
|
|
`softdog` at the time, which cannot rescue a hard freeze because the frozen
|
|
kernel is what would have to fire the timer; and `watchdog-mux` held the device
|
|
without ever arming it, because it only pets while an HA client is connected
|
|
and this cluster has no HA resources.
|
|
|
|
Applied and re-runnable via
|
|
[`playbooks/esh-pve-hardware-watchdog.yaml`](../../playbooks/esh-pve-hardware-watchdog.yaml).
|
|
|
|
⚠️ **Reverse this before configuring Proxmox HA on esh-pve** — HA fencing needs
|
|
`watchdog-mux` to own `/dev/watchdog`. Not a near-term concern: this is a
|
|
two-node cluster with no qdevice, so losing one node already costs quorum.
|
|
|
|
⚠️ The watchdog is armed but **has not been observed firing**. Confirming it
|
|
means deliberately wedging the host.
|
|
|
|
## vPro / AMT — not currently usable
|
|
|
|
The board is vPro-capable, but AMT needs the chipset-integrated Intel PHY (one
|
|
of the two i226 **RJ45** ports) and this host reaches the network only over
|
|
**SFP+** (Intel X710 → port 27 on the Garage switch), presenting a single MAC.
|
|
AMT cannot ride a discrete/SFP+ NIC. Cable an onboard RJ45 and provision AMT in
|
|
MEBx to get out-of-band power control; until then, recovery is the hardware
|
|
watchdog above or a physical trip.
|
|
|
|
## Refresh state
|
|
|
|
```bash
|
|
scripts/refresh-server-info.sh esh-pve
|
|
```
|
|
|
|
Same Proxmox-inspect caveat as `pfi-pve`.
|
|
|
|
## Placement rule
|
|
|
|
Hypervisor for ESH home-lab VMs. Not part of the PFI colo topology.
|
|
|
|
## ⚠ Cluster membership and the dark-tile gotcha
|
|
|
|
This node (`pve`) is half of the 2-node **`esh-pve-cluster`** with `esh-nas-pve`
|
|
(esh-pve-nas, 10.0.50.55). `Expected votes: 2`, `Quorum: 2`, no qdevice — so
|
|
**rebooting either node makes the survivor's `/etc/pve` read-only** until the
|
|
partner is back. Guests keep running; config changes do not. Plan reboots of
|
|
either node accordingly (`pvecm expected 1` on the survivor, or accept the
|
|
read-only window).
|
|
|
|
⚠ **A dark/greyed node tile usually means `pvestatd`, not a dead node.**
|
|
`pvestatd` SEGV'd here on **2026-05-28 and stayed dead for 82 days** — the node
|
|
was quorate and healthy the entire time, with `pve-cluster`, `corosync`,
|
|
`pvedaemon`, `pveproxy` and both HA services active and all three guests running.
|
|
It is only the *reporting* daemon, so its death is invisible except that the UI
|
|
has nothing to render. It has SEGV'd four times (2025-08-18, 2025-09-04,
|
|
2025-09-15, 2026-05-28) — treat a recurrence as expected, not novel.
|
|
|
|
systemctl reset-failed pvestatd && systemctl restart pvestatd
|
|
|
|
Restarted 2026-08-18. Worth a watchdog: nothing alerts when it dies, and the only
|
|
symptom is a cosmetic one nobody looks at for months.
|