esh-pfi-infrastructure/servers/esh-pve/README.md

# esh-pve

Proxmox VE hypervisor for the ESH home lab (`pve.esteban.net`). Non-PFI scope.

## Network

- **LAN IP:** 10.0.250.35
- **FQDN:** `pve.esteban.net`
- **Web UI:** `https://10.0.250.35:8006`
- **SSH:** `root@esh-pve` (key auth)

## Hardware

- **CPU:** 13th Gen Intel Core i9-13900H
- **RAM:** 62.5 GB
- **Kernel:** `6.8.12-42-pve`, PVE 8.4.20 (as of 2026-09-24)
- **GPU:** NVIDIA RTX 2000E Ada, 16 GB (`01:00.0`) — driven by the **host** (580.178.04, DKMS), used by the esh-ml1 LXC. See below.
- **Storage:** local `pve-root` at 94G (21% used) + two large NFS mounts from `10.0.50.50`:
  - `/mnt/pve/esh-nas` — 92 TB available on `/mnt/pvestore`
  - `/mnt/pve/tank-vmbu` — 93 TB, VM backup target (~1.4 TB used)

## What it runs

`esh-docker-vm` (`10.0.50.45`) is a VM here. Other home-lab VMs are not yet catalogued in this workspace — run `qm list` on the host for the live inventory.

VM **102 `esh-vm-workstation` is RETIRED (2026-09-24, Prime: "not the machine I
want running a virtual Windows machine").** `qm destroy 102 --purge` removed its
config and EFI disk. Its final backup, `pbs-ana:backup/vm/102/2026-09-10T10:32:20Z`
(verification ok), is **protected** against prune, so the retirement is
restorable. It had full GPU passthrough (`hostpci0: 0000:01:00,pcie=1,x-vga=1`)
and was the prime suspect in the 2026-08-19 freeze below.

⚠ **Found while retiring it: a hung VM-102 process since 2026-09-10 0332.** That
night's vzdump started the stopped VM with the T400 passed through to read its
disks, and the `kvm -id 102` process never exited. It is in **D state** (stuck in
`blkdev` teardown) together with its `swtpm` and a udev worker, so it cannot be
killed. It holds **16 GB of locked RAM**, the T400, and the two LVs `qm destroy`
could not remove (`pve/vm-102-disk-1` 256G, `pve/vm-102-disk-2` TPM). `qm status
102` said `stopped` the whole time. **Only a host reboot clears it.** It is the
same class of fault as the August freeze: VFIO plus this T400.

**Done 2026-09-24 2119–2137 (Prime on site).** Guests were shut down cleanly and
the host was powered off. The T400 was **removed** and replaced by an **NVIDIA RTX
2000 / 2000E Ada** (`01:00.0`, `10de:28b0`, subsystem `10de:1871`). The host was
back at 2136:49. The reboot cleared the hung `kvm -id 102`, and the two leftover
LVs were then removed; host RAM in use fell to ~4 GB before esh-vm-docker was
fully up. esh-vm-docker, esh-vm-db and esh-scale all started by themselves.
esh-vm-docker's NFS (`/mnt/books`, `/mnt/backup`) mounted, traefik served with no
403s, and AdGuard resolved.

The new card got its driver the same night — see the next section.

## GPU — host driver + the esh-ml1 LXC (2026-09-24)

**Prime's decision: LXC + host driver, NOT a VFIO VM.** Both hangs on this box
this year came from VFIO passthrough. The card now runs under the host's own
NVIDIA driver and is shared into **CT 110 `esh-ml1`**, which serves embedding
and reranking ([`servers/esh-ml1/README.md`](../esh-ml1/README.md)).

- **Driver 580.178.04, open kernel modules, DKMS**, from NVIDIA's `.run`
  (`/root/nvidia/`). Applied by
  [`playbooks/esh-pve-nvidia-host.yaml`](../../playbooks/esh-pve-nvidia-host.yaml)
  **live, with no reboot**: nouveau was never loaded and nothing held the card.
- **`nvidia-persistenced.service`** (ours, in `/etc/systemd/system`) runs
  `nvidia-modprobe -c0 -u` and the persistence daemon **before
  `pve-guests`**, so `/dev/nvidia0`, `nvidiactl` and `nvidia-uvm{,-tools}`
  exist when CT 110 starts. Without them `pct start` refuses the `devN:`
  entries. If the unit fails, only CT 110 fails; no other guest depends on it.
- `/etc/modules-load.d/nvidia.conf` loads `nvidia` + `nvidia_uvm`.
- **Retired:** the T400's vfio-pci id binding (`/etc/modprobe.d/vfio.conf`, moved
  to `/root/nvidia/vfio.conf.retired-2026-09-24`) and `blacklist nvidia` in
  `/etc/modprobe.d/blacklist.conf` (`blacklist nouveau` stays). The `vfio*`
  lines in `/etc/modules` were left alone; they bind nothing. The initramfs was
  not rebuilt, to keep the boot path untouched; the next kernel update will.
- The installer also enabled NVIDIA's `nvidia-{suspend,hibernate,resume}`
  units. They only act on a suspend and are harmless on a server.
- ⚠ **Kernel updates:** DKMS rebuilds the module when a new
  `proxmox-headers-6.8.*` arrives (the `proxmox-headers-6.8` meta-package is
  installed). Before opting into a different kernel series, install that
  series' headers meta-package, or the next boot comes up with no GPU and CT
  110 will not start.
- ⚠ **The host module and CT 110's libraries must stay the same version.**
  Upgrade both playbooks together (host first).

## Watchdog — hardware, not software

This host runs the **PCH hardware watchdog** (`iTCO_wdt`, 60 s, owned by
systemd via `RuntimeWatchdogSec`). `softdog` is blacklisted and Proxmox's
`watchdog-mux` is **masked**.

That is deliberate. On 2026-08-19 esh-pve hard-froze at 03:34 and stayed frozen
for ~4.5 hours until someone power-cycled it by hand — taking the only DNS
resolver the `esh-userland` VLAN is handed down with it. It was running
`softdog` at the time, which cannot rescue a hard freeze because the frozen
kernel is what would have to fire the timer; and `watchdog-mux` held the device
without ever arming it, because it only pets while an HA client is connected
and this cluster has no HA resources.

Applied and re-runnable via
[`playbooks/esh-pve-hardware-watchdog.yaml`](../../playbooks/esh-pve-hardware-watchdog.yaml).

⚠️ **Reverse this before configuring Proxmox HA on esh-pve** — HA fencing needs
`watchdog-mux` to own `/dev/watchdog`. Not a near-term concern: this is a
two-node cluster with no qdevice, so losing one node already costs quorum.

⚠️ The watchdog is armed but **has not been observed firing**. Confirming it
means deliberately wedging the host.

## vPro / AMT — not currently usable

The board is vPro-capable, but AMT needs the chipset-integrated Intel PHY (one
of the two i226 **RJ45** ports) and this host reaches the network only over
**SFP+** (Intel X710 → port 27 on the Garage switch), presenting a single MAC.
AMT cannot ride a discrete/SFP+ NIC. Cable an onboard RJ45 and provision AMT in
MEBx to get out-of-band power control; until then, recovery is the hardware
watchdog above or a physical trip.

## Refresh state

```bash
scripts/refresh-server-info.sh esh-pve
```

Same Proxmox-inspect caveat as `pfi-pve`.

## Placement rule

Hypervisor for ESH home-lab VMs. Not part of the PFI colo topology.

## ⚠ Cluster membership and the dark-tile gotcha

This node (`pve`) is half of the 2-node **`esh-pve-cluster`** with `esh-nas-pve`
(esh-pve-nas, 10.0.50.55). `Expected votes: 2`, `Quorum: 2`, no qdevice — so
**rebooting either node makes the survivor's `/etc/pve` read-only** until the
partner is back. Guests keep running; config changes do not. Plan reboots of
either node accordingly (`pvecm expected 1` on the survivor, or accept the
read-only window).

⚠ **A dark/greyed node tile usually means `pvestatd`, not a dead node.**
`pvestatd` SEGV'd here on **2026-05-28 and stayed dead for 82 days** — the node
was quorate and healthy the entire time, with `pve-cluster`, `corosync`,
`pvedaemon`, `pveproxy` and both HA services active and all three guests running.
It is only the *reporting* daemon, so its death is invisible except that the UI
has nothing to render. It has SEGV'd four times (2025-08-18, 2025-09-04,
2025-09-15, 2026-05-28) — treat a recurrence as expected, not novel.

    systemctl reset-failed pvestatd && systemctl restart pvestatd

Restarted 2026-08-18. Worth a watchdog: nothing alerts when it dies, and the only
symptom is a cosmetic one nobody looks at for months.