feat(esh-ml1): wire telemetry and monitoring; fix Dozzle's stale agent list

- Beszel: NVIDIA agent (stacks/beszel hosts/esh-ml1.yaml), hub system
  registered with Status/Disk/CPU/Memory/Temperature alerts; GPU samples
  verified (RTX 2000E Ada util, VRAM, power, temp).
- Uptime Kuma: health monitors for the TEI embed (:8001) and rerank (:8013)
  services — the sole backends behind the gateway, so the one exception to
  "seats are out of Kuma's lane". Reward seat excluded (no consumer).
- Homepage: dockerd on tcp/2375 bound to 10.0.50.80 (playbook step);
  esh-ml1-docker added to docker.yaml.
- Dozzle: agent v10.4.1 on esh-ml1. The hub's DOZZLE_REMOTE_AGENT still
  named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3; repointed
  to 10.251.50.54 / 10.6.110.50 (fv-ml1 and irv-ml1 logs were missing).
- Documented nh3-dev's Beszel agent failing on NH3 cold start (bind of an
  automounted NAS share); revived by hand, fix still open.
This commit is contained in:
vh
2026-09-25 09:20:36 -07:00
parent 52612cbe96
commit 65dc586497
9 changed files with 165 additions and 20 deletions
+16 -6
View File
@@ -117,13 +117,22 @@ no longer deployed sidecars here. See Recent decisions.)
_As of 2026-09-24 ~2245 PT._
### ⭐ TEI is the fleet embed/rerank engine; esh-ml1 is the SOLE backend (2026-09-25)
### ⭐ esh-ml1 = the fleet's utility GPU host (2026-09-25): embed/rerank (TEI) + reward (vLLM)
Cut-over done 0830 PT: gateway `qwen3-embedding` + `reranker` (+ DB alias a3) →
TEI on esh-ml1; fv-ml1's two vLLM seats retired. ⚠ **Single backend until the
second RTX 2000 arrives**, and **esh-ml1 has no monitoring yet**: wire Beszel +
Kuma next. Next conversation with Prime: "what else moves" (vllm-reward is the
candidate; **parakeet stays**). → Recent decisions.
TEI is the fleet embed/rerank engine (Prime) and esh-ml1 is the SOLE backend
(⚠ no failover until the second RTX 2000 arrives). The Skywork reward seat moved
there too; the audit found nothing supersedes it. fv-ml1 GPU 1 freed ~15 GB
across the day; only `vllm-coder` is left in `stacks/vllm`, and **parakeet stays**
on fv-ml1. Monitoring is wired: Beszel with GPU and 5 alerts, Kuma #27/#28,
Homepage cards, a Dozzle agent. Open follow-ups:
- **worldtree-dev**: Domari's Skywork path is broken (dead IP + wrong schema).
Told 2026-09-25; waiting on them.
- **nh3-dev Beszel agent** dies on every NH3 cold start (it binds an automounted
NAS share). Fix = drop its NAS extra-filesystems. Not done.
- **Dozzle nh3-docker agent** has been stopped by hand since ~2026-04. Revive or drop.
- **Beszel superuser password was echoed into this session's transcript**
(local only, not in memory). Rotation offered to Prime.
→ Recent decisions.
### ✅ esh-ml1 built (2026-09-24) — two follow-ups
@@ -169,6 +178,7 @@ a shutdown (stays off by design). Outcomes and revert in
## Recent decisions
- `[2026-09-25]` **Reward seat audited (Skywork-Reward-V2-Llama-3.1-8B still #1 of 188 on RewardBench 2; our AWQ ≈ bf16 within noise; double BOS costs ~2.7 pts) and moved fv-ml1 → esh-ml1; esh-ml1 monitoring wired (Beszel+GPU, Kuma, Homepage, Dozzle); Dozzle hub's stale agent IPs fixed; nh3-dev Beszel agent revived.** → `persistent-memory.d/2026-09-25-reward-move-and-esh-ml1-monitoring.md`
- `[2026-09-25]` **TEI adopted as the fleet embed/rerank engine (Prime); esh-ml1 is the sole gateway backend for `qwen3-embedding` + `reranker`; fv-ml1's vLLM embed/rerank seats retired (~6.1 GB freed); parakeet stays on fv-ml1.** → `persistent-memory.d/2026-09-25-tei-fleet-embed-rerank.md`
- `[2026-09-24]` **esh-ml1 built — RTX 2000E Ada as CT 110 on esh-pve (host driver 580.178.04 DKMS, loaded live, no reboot), vLLM embed+rerank at measured parity with fv-ml1, wired as LiteLLM `order: 2` failover; dead DB alias `reranker-a3-bge-v2-m3` (pre-relocation IP) repaired.** → `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md`
- `[2026-09-24]` **esh-pve: VM 102 retired, a 14-day hung VFIO process found and cleared, T400 → RTX 2000 Ada; Prime chose LXC + host driver for embed/rerank** (implementation deferred to next session, tracked here + `servers/esh-pve/README.md`). → `persistent-memory.d/2026-09-24-esh-pve-vm102-retired-gpu-swap.md`