docs(esh): record the 2-node cluster, and that a dark tile is pvestatd

Two findings from chasing node `pve` showing dark in the UI.

esh-pve and esh-pve-nas are a 2-node cluster (esh-pve-cluster, expected
votes 2, quorum 2, no qdevice). This was undocumented, and last night's
migration rebooted one of the members without accounting for it. Nothing
broke -- quorum is intact and both nodes report the same ring id -- but
that was luck. Rebooting either node drops the survivor below quorum and
makes its /etc/pve read-only until the partner returns. It matters for
the pending confirmation reboot and the 225-package upgrade, both of
which take a node down.

The dark tile itself was NOT last night's doing. pvestatd SEGV'd on
2026-05-28 and had been dead 82 days; the journal has nothing between
that crash and the restart today. It is only the reporting daemon, so
the node stayed quorate and healthy with all services active and all
three guests running the whole time -- the UI simply had nothing telling
it the node was alive. Fourth SEGV in that unit's history, so treat a
recurrence as expected and consider a watchdog: nothing alerts on it,
and the sole symptom is cosmetic enough to go unnoticed for months.
This commit is contained in:
vh
2026-08-18 06:43:02 -07:00
parent 5f11d1b3cb
commit b84f8a996f
2 changed files with 31 additions and 0 deletions
+9
View File
@@ -51,6 +51,15 @@ Media services on this box sit at `10.0.50.56` (Plex) and `10.0.50.57` (Jellyfin
- **Pools:** `nvme` (2× 931 GB NVMe mirror — 32 G used, 867 G free, holds every guest
rootfs), `ssd` (4× 894 GB Intel SATA, 2 mirrors — 1.42 T free), `tank`
(12× 14.6 TB raidz2 ×2 — 40 T of 175 T). Plus NFS `/mnt/pve/tank-vmbu` for VM backups.
- ⚠⚠ **THIS HOST IS HALF OF A 2-NODE PROXMOX CLUSTER** — `esh-pve-cluster`, nodes
`pve` (esh-pve, 10.0.250.35, nodeid 1) and `esh-nas-pve` (this host, nodeid 2).
`Expected votes: 2`, `Quorum: 2`, no qdevice. **Taking either node down drops the
survivor below quorum and makes its `/etc/pve` read-only** — guests keep running,
but no config change, no VM start/stop, no storage edit works until the partner
returns. This was undocumented and nearly bit the 2026-08-18 migration, which
rebooted this node without accounting for it. Before any planned reboot of either
node, either accept the read-only window on the survivor or set
`pvecm expected 1` on it for the duration. Discovered 2026-08-18.
- ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on THIS host and serves `hard` NFS.**
Never reboot this host casually — quiesce the clients first. Measured from the
server on 2026-08-18 (`ss -tn '( sport = :2049 )'` inside CT 103), there are