refactor(playbooks): host-generic GPU host + GPU LXC playbooks for nh3-ml1
- esh-pve-nvidia-host -> pve-nvidia-host: headers/dkms/build-essential step, nouveau blacklist + guarded unload (refuses if nouveau bound a device) - esh-ml1-lxc -> gpu-lxc: host vars have no defaults (elway aborts on undefined), rootfs storage/startup order parameterized, CT kept out of all-guests vzdump jobs - embed-rerank: Homepage labels take HOST_NAME/HOST_IP, defaults = esh-ml1
This commit is contained in:
@@ -111,14 +111,17 @@ free.
|
||||
|
||||
## How it is built
|
||||
|
||||
1. [`playbooks/esh-pve-nvidia-host.yaml`](../../playbooks/esh-pve-nvidia-host.yaml)
|
||||
1. [`playbooks/pve-nvidia-host.yaml`](../../playbooks/pve-nvidia-host.yaml)
|
||||
— driver **580.178.04** (open modules, DKMS) on the **hypervisor**, the
|
||||
`nvidia-persistenced` unit that creates the device nodes before
|
||||
`pve-guests`, and removal of the old VFIO/blacklist config.
|
||||
2. [`playbooks/esh-ml1-lxc.yaml`](../../playbooks/esh-ml1-lxc.yaml) — the CT,
|
||||
2. [`playbooks/gpu-lxc.yaml`](../../playbooks/gpu-lxc.yaml) — the CT,
|
||||
`dev0–3` GPU nodes, the NVIDIA userspace from the **same `.run`** with
|
||||
`--no-kernel-modules`, fleet ids (infra-ops 850, docker 851, vh 1000),
|
||||
docker-ce and nvidia-container-toolkit with `no-cgroups = true`.
|
||||
Host-generic since 2026-09-25 (it also builds nh3-ml1). Its host vars have
|
||||
**no defaults**, so pass esh-ml1's set: the exact `--var` line is in the
|
||||
playbook header under "Run (esh-ml1)".
|
||||
3. `scripts/deploy-stack.sh esh-ml1 embed-rerank`, then `docker compose up -d`.
|
||||
|
||||
Both playbooks are idempotent; re-run them to repair.
|
||||
@@ -142,7 +145,7 @@ build and this CT will fail to start at the next boot.
|
||||
|---|---|---|
|
||||
| **Beszel** (host + GPU telemetry) | NVIDIA agent `henrygd/beszel-agent-nvidia:0.18.7`, `stacks/beszel` + `hosts/esh-ml1.yaml`. Hub system `ridkfdpwfq3f730`. GPU util, VRAM, power and temperature are sampled. | alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m (GPU included) |
|
||||
| **Uptime Kuma** (service) | `Embed — Qwen3 0.6B (TEI, esh-ml1)` → `:8001/health` (#27); `Rerank — bge-v2-m3 (TEI, esh-ml1)` → `:8013/health` (#28). TEI's health runs the backend. | alerts → infra-ops via althing-alert-bridge; `stacks/uptimekuma/monitors.yaml` |
|
||||
| **Homepage** | three cards under *AI - Eval & Retrieval*. dockerd exposes tcp/2375 on `10.0.50.80` only (fleet norm, `playbooks/esh-ml1-lxc.yaml`). | `stacks/homepage/conf/docker.yaml` → `esh-ml1-docker` |
|
||||
| **Homepage** | three cards under *AI - Eval & Retrieval*. dockerd exposes tcp/2375 on `10.0.50.80` only (fleet norm, `playbooks/gpu-lxc.yaml`). | `stacks/homepage/conf/docker.yaml` → `esh-ml1-docker` |
|
||||
| **Dozzle** (logs) | agent `v10.4.1` on `10.0.50.80:7007`, compose dir `dozzle-agent` | hub on ana-docker :8088 |
|
||||
|
||||
The reward seat has **no Kuma check on purpose**: seats are outside Kuma's lane,
|
||||
|
||||
@@ -59,7 +59,7 @@ and reranking ([`servers/esh-ml1/README.md`](../esh-ml1/README.md)).
|
||||
|
||||
- **Driver 580.178.04, open kernel modules, DKMS**, from NVIDIA's `.run`
|
||||
(`/root/nvidia/`). Applied by
|
||||
[`playbooks/esh-pve-nvidia-host.yaml`](../../playbooks/esh-pve-nvidia-host.yaml)
|
||||
[`playbooks/pve-nvidia-host.yaml`](../../playbooks/pve-nvidia-host.yaml)
|
||||
**live, with no reboot**: nouveau was never loaded and nothing held the card.
|
||||
- **`nvidia-persistenced.service`** (ours, in `/etc/systemd/system`) runs
|
||||
`nvidia-modprobe -c0 -u` and the persistence daemon **before
|
||||
|
||||
Reference in New Issue
Block a user