From 65dc586497b38944e8065b821f37f11b041a2e63 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 25 Sep 2026 09:20:36 -0700 Subject: [PATCH] feat(esh-ml1): wire telemetry and monitoring; fix Dozzle's stale agent list MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Beszel: NVIDIA agent (stacks/beszel hosts/esh-ml1.yaml), hub system registered with Status/Disk/CPU/Memory/Temperature alerts; GPU samples verified (RTX 2000E Ada util, VRAM, power, temp). - Uptime Kuma: health monitors for the TEI embed (:8001) and rerank (:8013) services — the sole backends behind the gateway, so the one exception to "seats are out of Kuma's lane". Reward seat excluded (no consumer). - Homepage: dockerd on tcp/2375 bound to 10.0.50.80 (playbook step); esh-ml1-docker added to docker.yaml. - Dozzle: agent v10.4.1 on esh-ml1. The hub's DOZZLE_REMOTE_AGENT still named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3; repointed to 10.251.50.54 / 10.6.110.50 (fv-ml1 and irv-ml1 logs were missing). - Documented nh3-dev's Beszel agent failing on NH3 cold start (bind of an automounted NAS share); revived by hand, fix still open. --- ...9-25-reward-move-and-esh-ml1-monitoring.md | 55 +++++++++++++++++++ persistent-memory.md | 22 ++++++-- playbooks/esh-ml1-lxc.yaml | 22 ++++++++ servers/esh-ml1/README.md | 27 +++++---- stacks/beszel/README.md | 10 ++++ stacks/beszel/hosts/esh-ml1.yaml | 14 +++++ stacks/dozzle/README.md | 14 ++++- stacks/homepage/conf/docker.yaml | 6 ++ stacks/uptimekuma/monitors.yaml | 15 +++++ 9 files changed, 165 insertions(+), 20 deletions(-) create mode 100644 persistent-memory.d/2026-09-25-reward-move-and-esh-ml1-monitoring.md create mode 100644 stacks/beszel/hosts/esh-ml1.yaml diff --git a/persistent-memory.d/2026-09-25-reward-move-and-esh-ml1-monitoring.md b/persistent-memory.d/2026-09-25-reward-move-and-esh-ml1-monitoring.md new file mode 100644 index 0000000..a543acd --- /dev/null +++ b/persistent-memory.d/2026-09-25-reward-move-and-esh-ml1-monitoring.md @@ -0,0 +1,55 @@ +# `[2026-09-25]` Reward seat audit + move to esh-ml1; esh-ml1 monitoring wired + +**Prime:** "Audit the reward seat to see if something has superseded what we +deployed — then move it to esh-ml1 as well, then wire up telemetry and monitoring." + +**Audit (full record `stacks/reward-seat/README.md`).** +- Not superseded. Skywork-Reward-V2-Llama-3.1-8B is #1 of 188 in AllenAI's official + RewardBench 2 per-sample results (mean over the 5 non-Ties subsets, 84.7; the + dataset was last updated 2025-12-11). No Skywork V3 exists. The -40M sibling is + vendor-marked experimental. Caveat: 2026 models never submitted to RewardBench 2 + would not appear. +- Our local AWQ W4A16: 0.847 vs published bf16 0.860 on 150 prompts (±2.9 pt SE), + 96.2% pairwise agreement, Pearson 0.991. +- ⚠ vLLM adds a SECOND BOS to already-templated text unless + `add_special_tokens:false` is sent: −2.7 pts. That was a harness confound in the + first measurement (0.820), caught by checking `/tokenize`. +- **No working consumer.** 0 requests 2026-09-13 → 09-25. Worldtree Domari + hardcodes the dead `10.250.50.54:8003` and sends a non-vLLM schema + (`prompt`/`responses` → `rewards`), so it always fell back to Selene/chat-judge. + worldtree-dev told (single message; their call). + +**Move.** +- sha256-identical model copy. +- vLLM v0.24.0 on esh-ml1 `:8003`, util 0.55 (KV 21,328 tokens = 1.30× of a 16k + request). +- Parity vs fv-ml1 on 600 RewardBench 2 texts: + - 149/150 verdicts and 99.8% pairwise signs identical. + - Raw |Δ| median 0.049; p99 0.22 vs the self-noise p99 of 0.16–0.20. +- Gateway `/scalar-judge` → `10.0.50.80:8003` (verified 3/3). +- fv-ml1 `vllm-reward` removed; `stacks/vllm` now = `vllm-coder` only. + +**Monitoring.** +- **Beszel:** NVIDIA agent (GPU util/VRAM/power/temp), system + `ridkfdpwfq3f730`, 5 alerts incl. Temperature >85 °C. +- **Kuma:** #27 embed, #28 rerank (the only seats inside Kuma's lane, because they + are the sole backends). Reward excluded (no consumer). +- **Homepage:** dockerd tcp/2375 on 10.0.50.80, which restarted docker (all + services healthy within ~31 s), and `esh-ml1-docker` in docker.yaml. Three cards + appeared without a Homepage restart. +- **Dozzle:** agent v10.4.1 pinned to the hub's version. It needed an empty + `traefik-net` network. + +**Found while wiring (same "rots on re-IP" class as the a3 alias):** +- The Dozzle hub still listed ana-ml2 `10.250.50.54` and irv-ml1's retired + `10.100.79.3`. fv-ml1 and irv-ml1 logs had been missing since their moves. Fixed; + the hub now reports 7 clients. +- nh3-dev's Beszel agent had been down since the 09-24 NH3 power recovery (bind of + the automounted `/mnt/smithy` failed at boot; docker never retries a failed + create). Started; the Beszel "up" alert reached althing, an end-to-end proof of + the alert route. Recurs on the next cold start until fixed. +- The nh3-docker Dozzle agent has been stopped by hand for ~5 months. Left alone. + +⚠ While checking the Beszel credential's shape, its password was printed into this +session's transcript. The transcript stays on nh3-dev and it was not written to any +memory; rotation offered to Prime. diff --git a/persistent-memory.md b/persistent-memory.md index 0d78b81..0e10698 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -117,13 +117,22 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-09-24 ~2245 PT._ -### ⭐ TEI is the fleet embed/rerank engine; esh-ml1 is the SOLE backend (2026-09-25) +### ⭐ esh-ml1 = the fleet's utility GPU host (2026-09-25): embed/rerank (TEI) + reward (vLLM) -Cut-over done 0830 PT: gateway `qwen3-embedding` + `reranker` (+ DB alias a3) → -TEI on esh-ml1; fv-ml1's two vLLM seats retired. ⚠ **Single backend until the -second RTX 2000 arrives**, and **esh-ml1 has no monitoring yet**: wire Beszel + -Kuma next. Next conversation with Prime: "what else moves" (vllm-reward is the -candidate; **parakeet stays**). → Recent decisions. +TEI is the fleet embed/rerank engine (Prime) and esh-ml1 is the SOLE backend +(⚠ no failover until the second RTX 2000 arrives). The Skywork reward seat moved +there too; the audit found nothing supersedes it. fv-ml1 GPU 1 freed ~15 GB +across the day; only `vllm-coder` is left in `stacks/vllm`, and **parakeet stays** +on fv-ml1. Monitoring is wired: Beszel with GPU and 5 alerts, Kuma #27/#28, +Homepage cards, a Dozzle agent. Open follow-ups: +- **worldtree-dev**: Domari's Skywork path is broken (dead IP + wrong schema). + Told 2026-09-25; waiting on them. +- **nh3-dev Beszel agent** dies on every NH3 cold start (it binds an automounted + NAS share). Fix = drop its NAS extra-filesystems. Not done. +- **Dozzle nh3-docker agent** has been stopped by hand since ~2026-04. Revive or drop. +- **Beszel superuser password was echoed into this session's transcript** + (local only, not in memory). Rotation offered to Prime. +→ Recent decisions. ### ✅ esh-ml1 built (2026-09-24) — two follow-ups @@ -169,6 +178,7 @@ a shutdown (stays off by design). Outcomes and revert in ## Recent decisions +- `[2026-09-25]` **Reward seat audited (Skywork-Reward-V2-Llama-3.1-8B still #1 of 188 on RewardBench 2; our AWQ ≈ bf16 within noise; double BOS costs ~2.7 pts) and moved fv-ml1 → esh-ml1; esh-ml1 monitoring wired (Beszel+GPU, Kuma, Homepage, Dozzle); Dozzle hub's stale agent IPs fixed; nh3-dev Beszel agent revived.** → `persistent-memory.d/2026-09-25-reward-move-and-esh-ml1-monitoring.md` - `[2026-09-25]` **TEI adopted as the fleet embed/rerank engine (Prime); esh-ml1 is the sole gateway backend for `qwen3-embedding` + `reranker`; fv-ml1's vLLM embed/rerank seats retired (~6.1 GB freed); parakeet stays on fv-ml1.** → `persistent-memory.d/2026-09-25-tei-fleet-embed-rerank.md` - `[2026-09-24]` **esh-ml1 built — RTX 2000E Ada as CT 110 on esh-pve (host driver 580.178.04 DKMS, loaded live, no reboot), vLLM embed+rerank at measured parity with fv-ml1, wired as LiteLLM `order: 2` failover; dead DB alias `reranker-a3-bge-v2-m3` (pre-relocation IP) repaired.** → `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md` - `[2026-09-24]` **esh-pve: VM 102 retired, a 14-day hung VFIO process found and cleared, T400 → RTX 2000 Ada; Prime chose LXC + host driver for embed/rerank** (implementation deferred to next session, tracked here + `servers/esh-pve/README.md`). → `persistent-memory.d/2026-09-24-esh-pve-vm102-retired-gpu-swap.md` diff --git a/playbooks/esh-ml1-lxc.yaml b/playbooks/esh-ml1-lxc.yaml index 1055b97..20042cb 100644 --- a/playbooks/esh-ml1-lxc.yaml +++ b/playbooks/esh-ml1-lxc.yaml @@ -32,6 +32,7 @@ vars: ctid: 110 hostname: esh-ml1 ip_cidr: 10.0.50.80/24 + ip_addr: 10.0.50.80 gateway: 10.0.50.1 vlan: 50 cores: 6 @@ -156,6 +157,27 @@ steps: EOF when: "! pct exec {{ ctid }} -- sh -c 'grep -q nvidia /etc/docker/daemon.json && grep -Eq \"^no-cgroups *= *true\" /etc/nvidia-container-runtime/config.toml' 2>/dev/null" + # Fleet norm: Homepage (on esh-docker-vm) discovers labelled containers by + # reading every host's Docker API on tcp/2375 (stacks/homepage/conf/docker.yaml). + # Same unauthenticated plaintext exposure as fv-ml1 and esh-docker-vm, bound to + # this CT's one address. ⚠ Restarting dockerd restarts every container here, + # the fleet embed/rerank service included (~5 s for TEI, ~60 s for the reward seat). + - name: Expose the Docker API on tcp/2375 for Homepage discovery + shell: | + pct exec {{ ctid }} -- bash -s <<'EOF' + set -euo pipefail + install -d /etc/systemd/system/docker.service.d + cat > /etc/systemd/system/docker.service.d/override.conf <<'EOC' + # Homepage discovery — see eshpfi playbooks/esh-ml1-lxc.yaml + [Service] + ExecStart= + ExecStart=/usr/bin/dockerd -H fd:// -H tcp://{{ ip_addr }}:2375 --containerd=/run/containerd/containerd.sock + EOC + systemctl daemon-reload + systemctl restart docker + EOF + when: "! pct exec {{ ctid }} -- grep -q 'tcp://' /etc/systemd/system/docker.service.d/override.conf 2>/dev/null" + verify: - name: Container is running with onboot set, started after the core guests shell: "pct status {{ ctid }} | grep -q running && pct config {{ ctid }} | grep -q '^onboot: 1' && pct config {{ ctid }} | grep -q '^startup: order=30'" diff --git a/servers/esh-ml1/README.md b/servers/esh-ml1/README.md index 9434fad..fcf4cec 100644 --- a/servers/esh-ml1/README.md +++ b/servers/esh-ml1/README.md @@ -136,17 +136,20 @@ for each new kernel). Moving esh-pve to a different kernel series (6.14 opt-in) needs that series' headers meta-package installed first, or the module will not build and this CT will fail to start at the next boot. -## Not yet wired +## Monitoring and telemetry (wired 2026-09-25) -⚠ **esh-ml1 became load-bearing on 2026-09-25, so these are no longer -optional.** Nothing alerts if it dies today. The gateway will just start -returning errors for `qwen3-embedding` and `reranker`. +| layer | what | where | +|---|---|---| +| **Beszel** (host + GPU telemetry) | NVIDIA agent `henrygd/beszel-agent-nvidia:0.18.7`, `stacks/beszel` + `hosts/esh-ml1.yaml`. Hub system `ridkfdpwfq3f730`. GPU util, VRAM, power and temperature are sampled. | alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m (GPU included) | +| **Uptime Kuma** (service) | `Embed — Qwen3 0.6B (TEI, esh-ml1)` → `:8001/health` (#27); `Rerank — bge-v2-m3 (TEI, esh-ml1)` → `:8013/health` (#28). TEI's health runs the backend. | alerts → infra-ops via althing-alert-bridge; `stacks/uptimekuma/monitors.yaml` | +| **Homepage** | three cards under *AI - Eval & Retrieval*. dockerd exposes tcp/2375 on `10.0.50.80` only (fleet norm, `playbooks/esh-ml1-lxc.yaml`). | `stacks/homepage/conf/docker.yaml` → `esh-ml1-docker` | +| **Dozzle** (logs) | agent `v10.4.1` on `10.0.50.80:7007`, compose dir `dozzle-agent` | hub on ana-docker :8088 | -- **Beszel**: no agent yet. -- **Uptime Kuma**: no check on `:8001/health` / `:8013/health` yet. -- **Homepage**: the compose carries labels, but esh-ml1 is not in - `stacks/homepage/conf/docker.yaml` (it would need dockerd on tcp/2375 like - the other hosts). -- **Path:** every consumer reaches esh-ml1 through the gateway at ana-docker, so - ESH callers hairpin ESH → Anaheim → ESH over the mesh. A mesh outage cuts - every consumer off, the ones at ESH included. +The reward seat has **no Kuma check on purpose**: seats are outside Kuma's lane, +and this one has no working consumer. Beszel and Homepage cover it. + +⚠ Disk: rootfs is ~66% used (vLLM image ~30 GB, TEI ~8 GB). The alert fires at 85%. + +**Path:** every consumer reaches esh-ml1 through the gateway at ana-docker, so +ESH callers hairpin ESH → Anaheim → ESH over the mesh. A mesh outage cuts every +consumer off, the ones at ESH included. diff --git a/stacks/beszel/README.md b/stacks/beszel/README.md index 36a1fee..df1e708 100644 --- a/stacks/beszel/README.md +++ b/stacks/beszel/README.md @@ -23,6 +23,16 @@ filesystem samples verified; fleet 13/14 up with known fv-ml1 outage. | irv-ml1 | beszel-agent-irv | /worktank, /storetank, /mnt/smithy | | vm-esh-nas | beszel-agent-esh-nas | /mnt/books, /mnt/share, /mnt/music, /mnt/media | | nh3-dev | beszel | /mnt/backup, /mnt/smithy | +| esh-ml1 (added 2026-09-25) | beszel | none — NVIDIA image (`hosts/esh-ml1.yaml`) for the RTX 2000E Ada | + +⚠ **nh3-dev's agent was DOWN from the 2026-09-24 NH3 power recovery until +2026-09-25.** Docker could not bind `/mnt/smithy` at boot ("no such device"): since +`1cbde50` the NAS shares are automounted, and nh3-nas was not up yet. Docker does not +retry a container that fails to *create*, so `unless-stopped` never brought it back. +Started by hand. **It will recur on the next NH3 cold start** until the agent stops +depending on NAS mounts at boot. The NAS capacity is already reported by +nh3-nas's own agent, so dropping the two extra filesystems here is the simplest fix. +Open item. Use `infra-ops@` with passwordless sudo, except vm-esh-nas: `lkraven@10.0.50.154` has Docker access. Irvine's hub address is diff --git a/stacks/beszel/hosts/esh-ml1.yaml b/stacks/beszel/hosts/esh-ml1.yaml new file mode 100644 index 0000000..3f9a0c9 --- /dev/null +++ b/stacks/beszel/hosts/esh-ml1.yaml @@ -0,0 +1,14 @@ +# esh-ml1 (CT 110 on esh-pve) — Beszel agent with NVIDIA GPU telemetry for the +# RTX 2000E Ada (utilization, VRAM, temperature, power). Docker-in-LXC: the +# NVIDIA container toolkit runs with no-cgroups=true (playbooks/esh-ml1-lxc.yaml). +# No extra filesystems: the root filesystem holds everything, models included. +services: + beszel-agent: + image: henrygd/beszel-agent-nvidia:0.18.7 + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: all + capabilities: [utility] diff --git a/stacks/dozzle/README.md b/stacks/dozzle/README.md index a5a9472..b03b7fc 100644 --- a/stacks/dozzle/README.md +++ b/stacks/dozzle/README.md @@ -3,9 +3,19 @@ Container log viewer. One UI on **ana-docker** aggregates logs from every Docker host via remote agents. **Deploys to:** -- **ana-docker** (hub) — UI at `http://10.250.50.70:8088` +- **ana-docker** (hub) — UI at `http://10.250.50.70:8088`; live compose dir `/opt/docker/compose/dozzle-hub` - **fv-ml1** (agent) — listens on `10.251.50.54:7007` -- **nh3-docker** (agent, cross-site) — listens on `10.100.50.40:7007` +- **esh-docker-vm** / **vm-esh-nas** (agents) — `10.0.50.45:7007`, `10.0.50.154:7007` +- **irv-ml1** (agent) — `10.6.110.50:7007` (mesh address) +- **esh-ml1** (agent, added 2026-09-25) — `10.0.50.80:7007`, pinned `v10.4.1` = the hub's version; compose dir `dozzle-agent`. The host needed an (empty) `traefik-net` network because the compose declares it external. +- **nh3-docker** (agent, cross-site) — `10.100.50.40:7007`. ⚠ **Stopped by hand ~2026-04 (Exited 0) and left that way**; the hub logs a refused connection for it. Revive it or drop it from the list deliberately. + +⚠ **The hub's agent list silently rots when a host moves.** Until 2026-09-25 it +still named ana-ml2's `10.250.50.54` and irv-ml1's retired `10.100.79.3`, so fv-ml1 +and irv-ml1 logs had been missing from Dozzle since their moves (2026-09-12 and +2026-09-06). Nothing reported it. After any re-IP, check the hub's +`DOZZLE_REMOTE_AGENT`, and check that `docker logs dozzle` shows `"clients":N` equal to +reachable agents + 1. - **corviduo-dev** (agent) — listens on `10.250.50.152:7007`. Compose at `/home/vh/docker/compose/dozzle-agent/` (not `/opt/docker/compose/` — see `servers/corviduo-dev/README.md` for why) One compose.yaml lives on each host. The per-host `.env` sets `COMPOSE_PROFILES=hub` or `COMPOSE_PROFILES=agent` so `docker compose up -d` brings up the right service. On the hub, add every agent to `DOZZLE_REMOTE_AGENT` as a comma-separated list (e.g. `10.251.50.54:7007,10.100.50.40:7007`). diff --git a/stacks/homepage/conf/docker.yaml b/stacks/homepage/conf/docker.yaml index 75be74b..6fdef26 100644 --- a/stacks/homepage/conf/docker.yaml +++ b/stacks/homepage/conf/docker.yaml @@ -31,6 +31,12 @@ irv-ml1-docker: host: 10.6.110.50 port: 2375 +# esh-ml1 — CT 110 on esh-pve: the fleet embed/rerank (TEI) + reward seats. +# dockerd listens only on its own address (playbooks/esh-ml1-lxc.yaml), 2026-09-25. +esh-ml1-docker: + host: 10.0.50.80 + port: 2375 + # Example TLS socket (if/when a host moves off plaintext 2375): # ana-pfi-docker: # host: 10.250.50.70 diff --git a/stacks/uptimekuma/monitors.yaml b/stacks/uptimekuma/monitors.yaml index e75ab97..1361cfe 100644 --- a/stacks/uptimekuma/monitors.yaml +++ b/stacks/uptimekuma/monitors.yaml @@ -92,6 +92,21 @@ monitors: - name: Asset Engine url: http://10.250.50.70:8200/api/v1/services + # ---- fleet embed/rerank: the one EXCEPTION to "seats are OUT" ---- + # Since 2026-09-25 these are the SOLE backends behind the gateway's + # `qwen3-embedding` and `reranker` (TEI on esh-ml1, no failover until the second + # RTX 2000 arrives). They are not come-and-go seats: dead = Worldtree recall, + # nevermore clustering and Open WebUI RAG all fail. TEI's /health runs the + # backend, so a loaded-but-broken model reads DOWN, not UP. The reward seat on the + # same box stays OUT: it has no working consumer (stacks/reward-seat/README.md). + - name: Embed — Qwen3 0.6B (TEI, esh-ml1) + url: http://10.0.50.80:8001/health + description: sole backend for gateway `qwen3-embedding` + + - name: Rerank — bge-v2-m3 (TEI, esh-ml1) + url: http://10.0.50.80:8013/health + description: sole backend for gateway `reranker` + - name: talk url: https://talk.nh3.phasefinal.com:8092/ description: fleet voice bench