vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
6b9ce5b14cc8061e8b9b7c17c50139cb45c3c930
esh-pfi-infrastructure/servers/esh-pve/README.md
T
vh 5402568b76 feat(esh-ml1): RTX 2000E Ada on esh-pve serves embed + rerank as a LiteLLM failover 
esh-pve: NVIDIA 580.178.04 (open modules, DKMS) installed on the host from
NVIDIA's .run and loaded live, no reboot. nvidia-persistenced unit creates the
device nodes before pve-guests; the T400's vfio-pci ids and `blacklist nvidia`
retired. playbooks/esh-pve-nvidia-host.yaml.

esh-ml1: CT 110, unprivileged Debian 12, 10.0.50.80, GPU nodes via devN,
NVIDIA userspace from the same .run (--no-kernel-modules), docker-ce +
nvidia-container-toolkit (no-cgroups). playbooks/esh-ml1-lxc.yaml. Not in the
vzdump job on purpose. DNS esh-ml1.esh.internal.

stacks/embed-rerank: Qwen3-Embedding-0.6B :8001 + bge-reranker-v2-m3 :8013 on
the same vLLM v0.24.0 digest and flags as fv-ml1. Measured parity: embed
cosine FV-vs-ESH median 0.999908 (min 0.999772), inside both self-noise
floors; rerank max |delta| 0.000145 vs floor 0.000181, identical ranking.

litellm: qwen3-embedding and reranker gain an esh-ml1 deployment at order 2
behind fv-ml1 (order 1). Order fallback proven with throwaway groups:
refused primary +0.15 s, host-down primary ~18.7 s per call, dead-only 500.

Also: repaired the DB-only alias reranker-a3-bge-v2-m3, dead since the
fv-ml1 relocation (still named 10.250.50.54); documented the third
unkillable homepage wedge on esh-docker-vm.
2026-09-24 22:38:55 -07:00

7.6 KiB
Raw Blame History

esh-pve

Proxmox VE hypervisor for the ESH home lab (pve.esteban.net). Non-PFI scope.

Network

  • LAN IP: 10.0.250.35
  • FQDN: pve.esteban.net
  • Web UI: https://10.0.250.35:8006
  • SSH: root@esh-pve (key auth)

Hardware

  • CPU: 13th Gen Intel Core i9-13900H
  • RAM: 62.5 GB
  • Kernel: 6.8.12-42-pve, PVE 8.4.20 (as of 2026-09-24)
  • GPU: NVIDIA RTX 2000E Ada, 16 GB (01:00.0) — driven by the host (580.178.04, DKMS), used by the esh-ml1 LXC. See below.
  • Storage: local pve-root at 94G (21% used) + two large NFS mounts from 10.0.50.50:
    • /mnt/pve/esh-nas — 92 TB available on /mnt/pvestore
    • /mnt/pve/tank-vmbu — 93 TB, VM backup target (~1.4 TB used)

What it runs

esh-docker-vm (10.0.50.45) is a VM here. Other home-lab VMs are not yet catalogued in this workspace — run qm list on the host for the live inventory.

VM 102 esh-vm-workstation is RETIRED (2026-09-24, Prime: "not the machine I want running a virtual Windows machine"). qm destroy 102 --purge removed its config and EFI disk. Its final backup, pbs-ana:backup/vm/102/2026-09-10T10:32:20Z (verification ok), is protected against prune, so the retirement is restorable. It had full GPU passthrough (hostpci0: 0000:01:00,pcie=1,x-vga=1) and was the prime suspect in the 2026-08-19 freeze below.

⚠ Found while retiring it: a hung VM-102 process since 2026-09-10 0332. That night's vzdump started the stopped VM with the T400 passed through to read its disks, and the kvm -id 102 process never exited. It is in D state (stuck in blkdev teardown) together with its swtpm and a udev worker, so it cannot be killed. It holds 16 GB of locked RAM, the T400, and the two LVs qm destroy could not remove (pve/vm-102-disk-1 256G, pve/vm-102-disk-2 TPM). qm status 102 said stopped the whole time. Only a host reboot clears it. It is the same class of fault as the August freeze: VFIO plus this T400.

Done 2026-09-24 2119–2137 (Prime on site). Guests were shut down cleanly and the host was powered off. The T400 was removed and replaced by an NVIDIA RTX 2000 / 2000E Ada (01:00.0, 10de:28b0, subsystem 10de:1871). The host was back at 2136:49. The reboot cleared the hung kvm -id 102, and the two leftover LVs were then removed; host RAM in use fell to ~4 GB before esh-vm-docker was fully up. esh-vm-docker, esh-vm-db and esh-scale all started by themselves. esh-vm-docker's NFS (/mnt/books, /mnt/backup) mounted, traefik served with no 403s, and AdGuard resolved.

The new card got its driver the same night — see the next section.

GPU — host driver + the esh-ml1 LXC (2026-09-24)

Prime's decision: LXC + host driver, NOT a VFIO VM. Both hangs on this box this year came from VFIO passthrough. The card now runs under the host's own NVIDIA driver and is shared into CT 110 esh-ml1, which serves embedding and reranking (servers/esh-ml1/README.md).

  • Driver 580.178.04, open kernel modules, DKMS, from NVIDIA's .run (/root/nvidia/). Applied by playbooks/esh-pve-nvidia-host.yaml live, with no reboot: nouveau was never loaded and nothing held the card.
  • nvidia-persistenced.service (ours, in /etc/systemd/system) runs nvidia-modprobe -c0 -u and the persistence daemon before pve-guests, so /dev/nvidia0, nvidiactl and nvidia-uvm{,-tools} exist when CT 110 starts. Without them pct start refuses the devN: entries. If the unit fails, only CT 110 fails; no other guest depends on it.
  • /etc/modules-load.d/nvidia.conf loads nvidia + nvidia_uvm.
  • Retired: the T400's vfio-pci id binding (/etc/modprobe.d/vfio.conf, moved to /root/nvidia/vfio.conf.retired-2026-09-24) and blacklist nvidia in /etc/modprobe.d/blacklist.conf (blacklist nouveau stays). The vfio* lines in /etc/modules were left alone; they bind nothing. The initramfs was not rebuilt, to keep the boot path untouched; the next kernel update will.
  • The installer also enabled NVIDIA's nvidia-{suspend,hibernate,resume} units. They only act on a suspend and are harmless on a server.
  • ⚠ Kernel updates: DKMS rebuilds the module when a new proxmox-headers-6.8.* arrives (the proxmox-headers-6.8 meta-package is installed). Before opting into a different kernel series, install that series' headers meta-package, or the next boot comes up with no GPU and CT 110 will not start.
  • ⚠ The host module and CT 110's libraries must stay the same version. Upgrade both playbooks together (host first).

Watchdog — hardware, not software

This host runs the PCH hardware watchdog (iTCO_wdt, 60 s, owned by systemd via RuntimeWatchdogSec). softdog is blacklisted and Proxmox's watchdog-mux is masked.

That is deliberate. On 2026-08-19 esh-pve hard-froze at 03:34 and stayed frozen for ~4.5 hours until someone power-cycled it by hand — taking the only DNS resolver the esh-userland VLAN is handed down with it. It was running softdog at the time, which cannot rescue a hard freeze because the frozen kernel is what would have to fire the timer; and watchdog-mux held the device without ever arming it, because it only pets while an HA client is connected and this cluster has no HA resources.

Applied and re-runnable via playbooks/esh-pve-hardware-watchdog.yaml.

⚠️ Reverse this before configuring Proxmox HA on esh-pve — HA fencing needs watchdog-mux to own /dev/watchdog. Not a near-term concern: this is a two-node cluster with no qdevice, so losing one node already costs quorum.

⚠️ The watchdog is armed but has not been observed firing. Confirming it means deliberately wedging the host.

vPro / AMT — not currently usable

The board is vPro-capable, but AMT needs the chipset-integrated Intel PHY (one of the two i226 RJ45 ports) and this host reaches the network only over SFP+ (Intel X710 → port 27 on the Garage switch), presenting a single MAC. AMT cannot ride a discrete/SFP+ NIC. Cable an onboard RJ45 and provision AMT in MEBx to get out-of-band power control; until then, recovery is the hardware watchdog above or a physical trip.

Refresh state

scripts/refresh-server-info.sh esh-pve

Same Proxmox-inspect caveat as pfi-pve.

Placement rule

Hypervisor for ESH home-lab VMs. Not part of the PFI colo topology.

⚠ Cluster membership and the dark-tile gotcha

This node (pve) is half of the 2-node esh-pve-cluster with esh-nas-pve (esh-pve-nas, 10.0.50.55). Expected votes: 2, Quorum: 2, no qdevice — so rebooting either node makes the survivor's /etc/pve read-only until the partner is back. Guests keep running; config changes do not. Plan reboots of either node accordingly (pvecm expected 1 on the survivor, or accept the read-only window).

⚠ A dark/greyed node tile usually means pvestatd, not a dead node. pvestatd SEGV'd here on 2026-05-28 and stayed dead for 82 days — the node was quorate and healthy the entire time, with pve-cluster, corosync, pvedaemon, pveproxy and both HA services active and all three guests running. It is only the reporting daemon, so its death is invisible except that the UI has nothing to render. It has SEGV'd four times (2025-08-18, 2025-09-04, 2025-09-15, 2026-05-28) — treat a recurrence as expected, not novel.

systemctl reset-failed pvestatd && systemctl restart pvestatd

Restarted 2026-08-18. Worth a watchdog: nothing alerts when it dies, and the only symptom is a cosmetic one nobody looks at for months.

Reference in New Issue View Git Blame Copy Permalink