feat(esh-ml1): wire telemetry and monitoring; fix Dozzle's stale agent list
- Beszel: NVIDIA agent (stacks/beszel hosts/esh-ml1.yaml), hub system registered with Status/Disk/CPU/Memory/Temperature alerts; GPU samples verified (RTX 2000E Ada util, VRAM, power, temp). - Uptime Kuma: health monitors for the TEI embed (:8001) and rerank (:8013) services — the sole backends behind the gateway, so the one exception to "seats are out of Kuma's lane". Reward seat excluded (no consumer). - Homepage: dockerd on tcp/2375 bound to 10.0.50.80 (playbook step); esh-ml1-docker added to docker.yaml. - Dozzle: agent v10.4.1 on esh-ml1. The hub's DOZZLE_REMOTE_AGENT still named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3; repointed to 10.251.50.54 / 10.6.110.50 (fv-ml1 and irv-ml1 logs were missing). - Documented nh3-dev's Beszel agent failing on NH3 cold start (bind of an automounted NAS share); revived by hand, fix still open.
This commit is contained in:
+16
-6
@@ -117,13 +117,22 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
_As of 2026-09-24 ~2245 PT._
|
||||
|
||||
### ⭐ TEI is the fleet embed/rerank engine; esh-ml1 is the SOLE backend (2026-09-25)
|
||||
### ⭐ esh-ml1 = the fleet's utility GPU host (2026-09-25): embed/rerank (TEI) + reward (vLLM)
|
||||
|
||||
Cut-over done 0830 PT: gateway `qwen3-embedding` + `reranker` (+ DB alias a3) →
|
||||
TEI on esh-ml1; fv-ml1's two vLLM seats retired. ⚠ **Single backend until the
|
||||
second RTX 2000 arrives**, and **esh-ml1 has no monitoring yet**: wire Beszel +
|
||||
Kuma next. Next conversation with Prime: "what else moves" (vllm-reward is the
|
||||
candidate; **parakeet stays**). → Recent decisions.
|
||||
TEI is the fleet embed/rerank engine (Prime) and esh-ml1 is the SOLE backend
|
||||
(⚠ no failover until the second RTX 2000 arrives). The Skywork reward seat moved
|
||||
there too; the audit found nothing supersedes it. fv-ml1 GPU 1 freed ~15 GB
|
||||
across the day; only `vllm-coder` is left in `stacks/vllm`, and **parakeet stays**
|
||||
on fv-ml1. Monitoring is wired: Beszel with GPU and 5 alerts, Kuma #27/#28,
|
||||
Homepage cards, a Dozzle agent. Open follow-ups:
|
||||
- **worldtree-dev**: Domari's Skywork path is broken (dead IP + wrong schema).
|
||||
Told 2026-09-25; waiting on them.
|
||||
- **nh3-dev Beszel agent** dies on every NH3 cold start (it binds an automounted
|
||||
NAS share). Fix = drop its NAS extra-filesystems. Not done.
|
||||
- **Dozzle nh3-docker agent** has been stopped by hand since ~2026-04. Revive or drop.
|
||||
- **Beszel superuser password was echoed into this session's transcript**
|
||||
(local only, not in memory). Rotation offered to Prime.
|
||||
→ Recent decisions.
|
||||
|
||||
### ✅ esh-ml1 built (2026-09-24) — two follow-ups
|
||||
|
||||
@@ -169,6 +178,7 @@ a shutdown (stays off by design). Outcomes and revert in
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-25]` **Reward seat audited (Skywork-Reward-V2-Llama-3.1-8B still #1 of 188 on RewardBench 2; our AWQ ≈ bf16 within noise; double BOS costs ~2.7 pts) and moved fv-ml1 → esh-ml1; esh-ml1 monitoring wired (Beszel+GPU, Kuma, Homepage, Dozzle); Dozzle hub's stale agent IPs fixed; nh3-dev Beszel agent revived.** → `persistent-memory.d/2026-09-25-reward-move-and-esh-ml1-monitoring.md`
|
||||
- `[2026-09-25]` **TEI adopted as the fleet embed/rerank engine (Prime); esh-ml1 is the sole gateway backend for `qwen3-embedding` + `reranker`; fv-ml1's vLLM embed/rerank seats retired (~6.1 GB freed); parakeet stays on fv-ml1.** → `persistent-memory.d/2026-09-25-tei-fleet-embed-rerank.md`
|
||||
- `[2026-09-24]` **esh-ml1 built — RTX 2000E Ada as CT 110 on esh-pve (host driver 580.178.04 DKMS, loaded live, no reboot), vLLM embed+rerank at measured parity with fv-ml1, wired as LiteLLM `order: 2` failover; dead DB alias `reranker-a3-bge-v2-m3` (pre-relocation IP) repaired.** → `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md`
|
||||
- `[2026-09-24]` **esh-pve: VM 102 retired, a 14-day hung VFIO process found and cleared, T400 → RTX 2000 Ada; Prime chose LXC + host driver for embed/rerank** (implementation deferred to next session, tracked here + `servers/esh-pve/README.md`). → `persistent-memory.d/2026-09-24-esh-pve-vm102-retired-gpu-swap.md`
|
||||
|
||||
Reference in New Issue
Block a user