refactor(playbooks): host-generic GPU host + GPU LXC playbooks for nh3-ml1

- esh-pve-nvidia-host -> pve-nvidia-host: headers/dkms/build-essential step,
  nouveau blacklist + guarded unload (refuses if nouveau bound a device)
- esh-ml1-lxc -> gpu-lxc: host vars have no defaults (elway aborts on undefined),
  rootfs storage/startup order parameterized, CT kept out of all-guests vzdump jobs
- embed-rerank: Homepage labels take HOST_NAME/HOST_IP, defaults = esh-ml1
This commit is contained in:
vh
2026-09-25 14:17:31 -07:00
parent cdd7605e89
commit bc278d4ba8
10 changed files with 170 additions and 72 deletions
+6 -3
View File
@@ -111,14 +111,17 @@ free.
## How it is built
1. [`playbooks/esh-pve-nvidia-host.yaml`](../../playbooks/esh-pve-nvidia-host.yaml)
1. [`playbooks/pve-nvidia-host.yaml`](../../playbooks/pve-nvidia-host.yaml)
— driver **580.178.04** (open modules, DKMS) on the **hypervisor**, the
`nvidia-persistenced` unit that creates the device nodes before
`pve-guests`, and removal of the old VFIO/blacklist config.
2. [`playbooks/esh-ml1-lxc.yaml`](../../playbooks/esh-ml1-lxc.yaml) — the CT,
2. [`playbooks/gpu-lxc.yaml`](../../playbooks/gpu-lxc.yaml) — the CT,
`dev0–3` GPU nodes, the NVIDIA userspace from the **same `.run`** with
`--no-kernel-modules`, fleet ids (infra-ops 850, docker 851, vh 1000),
docker-ce and nvidia-container-toolkit with `no-cgroups = true`.
Host-generic since 2026-09-25 (it also builds nh3-ml1). Its host vars have
**no defaults**, so pass esh-ml1's set: the exact `--var` line is in the
playbook header under "Run (esh-ml1)".
3. `scripts/deploy-stack.sh esh-ml1 embed-rerank`, then `docker compose up -d`.
Both playbooks are idempotent; re-run them to repair.
@@ -142,7 +145,7 @@ build and this CT will fail to start at the next boot.
|---|---|---|
| **Beszel** (host + GPU telemetry) | NVIDIA agent `henrygd/beszel-agent-nvidia:0.18.7`, `stacks/beszel` + `hosts/esh-ml1.yaml`. Hub system `ridkfdpwfq3f730`. GPU util, VRAM, power and temperature are sampled. | alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m (GPU included) |
| **Uptime Kuma** (service) | `Embed — Qwen3 0.6B (TEI, esh-ml1)` → `:8001/health` (#27); `Rerank — bge-v2-m3 (TEI, esh-ml1)` → `:8013/health` (#28). TEI's health runs the backend. | alerts → infra-ops via althing-alert-bridge; `stacks/uptimekuma/monitors.yaml` |
| **Homepage** | three cards under *AI - Eval & Retrieval*. dockerd exposes tcp/2375 on `10.0.50.80` only (fleet norm, `playbooks/esh-ml1-lxc.yaml`). | `stacks/homepage/conf/docker.yaml` → `esh-ml1-docker` |
| **Homepage** | three cards under *AI - Eval & Retrieval*. dockerd exposes tcp/2375 on `10.0.50.80` only (fleet norm, `playbooks/gpu-lxc.yaml`). | `stacks/homepage/conf/docker.yaml` → `esh-ml1-docker` |
| **Dozzle** (logs) | agent `v10.4.1` on `10.0.50.80:7007`, compose dir `dozzle-agent` | hub on ana-docker :8088 |
The reward seat has **no Kuma check on purpose**: seats are outside Kuma's lane,
+1 -1
View File
@@ -59,7 +59,7 @@ and reranking ([`servers/esh-ml1/README.md`](../esh-ml1/README.md)).
- **Driver 580.178.04, open kernel modules, DKMS**, from NVIDIA's `.run`
(`/root/nvidia/`). Applied by
[`playbooks/esh-pve-nvidia-host.yaml`](../../playbooks/esh-pve-nvidia-host.yaml)
[`playbooks/pve-nvidia-host.yaml`](../../playbooks/pve-nvidia-host.yaml)
**live, with no reboot**: nouveau was never loaded and nothing held the card.
- **`nvidia-persistenced.service`** (ours, in `/etc/systemd/system`) runs
`nvidia-modprobe -c0 -u` and the persistence daemon **before