feat(esh-ml1): wire telemetry and monitoring; fix Dozzle's stale agent list

- Beszel: NVIDIA agent (stacks/beszel hosts/esh-ml1.yaml), hub system registered with Status/Disk/CPU/Memory/Temperature alerts; GPU samples verified (RTX 2000E Ada util, VRAM, power, temp). - Uptime Kuma: health monitors for the TEI embed (:8001) and rerank (:8013) services — the sole backends behind the gateway, so the one exception to "seats are out of Kuma's lane". Reward seat excluded (no consumer). - Homepage: dockerd on tcp/2375 bound to 10.0.50.80 (playbook step); esh-ml1-docker added to docker.yaml. - Dozzle: agent v10.4.1 on esh-ml1. The hub's DOZZLE_REMOTE_AGENT still named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3; repointed to 10.251.50.54 / 10.6.110.50 (fv-ml1 and irv-ml1 logs were missing). - Documented nh3-dev's Beszel agent failing on NH3 cold start (bind of an automounted NAS share); revived by hand, fix still open.
2026-09-25 09:20:36 -07:00
parent 52612cbe96
commit 65dc586497
9 changed files with 165 additions and 20 deletions
@@ -0,0 +1,55 @@
+# `[2026-09-25]` Reward seat audit + move to esh-ml1; esh-ml1 monitoring wired
+
+**Prime:** "Audit the reward seat to see if something has superseded what we
+deployed — then move it to esh-ml1 as well, then wire up telemetry and monitoring."
+
+**Audit (full record `stacks/reward-seat/README.md`).**
+- Not superseded. Skywork-Reward-V2-Llama-3.1-8B is #1 of 188 in AllenAI's official
+  RewardBench 2 per-sample results (mean over the 5 non-Ties subsets, 84.7; the
+  dataset was last updated 2025-12-11). No Skywork V3 exists. The -40M sibling is
+  vendor-marked experimental. Caveat: 2026 models never submitted to RewardBench 2
+  would not appear.
+- Our local AWQ W4A16: 0.847 vs published bf16 0.860 on 150 prompts (±2.9 pt SE),
+  96.2% pairwise agreement, Pearson 0.991.
+- ⚠ vLLM adds a SECOND BOS to already-templated text unless
+  `add_special_tokens:false` is sent: −2.7 pts. That was a harness confound in the
+  first measurement (0.820), caught by checking `/tokenize`.
+- **No working consumer.** 0 requests 2026-09-13 → 09-25. Worldtree Domari
+  hardcodes the dead `10.250.50.54:8003` and sends a non-vLLM schema
+  (`prompt`/`responses` → `rewards`), so it always fell back to Selene/chat-judge.
+  worldtree-dev told (single message; their call).
+
+**Move.**
+- sha256-identical model copy.
+- vLLM v0.24.0 on esh-ml1 `:8003`, util 0.55 (KV 21,328 tokens = 1.30× of a 16k
+  request).
+- Parity vs fv-ml1 on 600 RewardBench 2 texts:
+  - 149/150 verdicts and 99.8% pairwise signs identical.
+  - Raw |Δ| median 0.049; p99 0.22 vs the self-noise p99 of 0.16–0.20.
+- Gateway `/scalar-judge` → `10.0.50.80:8003` (verified 3/3).
+- fv-ml1 `vllm-reward` removed; `stacks/vllm` now = `vllm-coder` only.
+
+**Monitoring.**
+- **Beszel:** NVIDIA agent (GPU util/VRAM/power/temp), system
+  `ridkfdpwfq3f730`, 5 alerts incl. Temperature >85 °C.
+- **Kuma:** #27 embed, #28 rerank (the only seats inside Kuma's lane, because they
+  are the sole backends). Reward excluded (no consumer).
+- **Homepage:** dockerd tcp/2375 on 10.0.50.80, which restarted docker (all
+  services healthy within ~31 s), and `esh-ml1-docker` in docker.yaml. Three cards
+  appeared without a Homepage restart.
+- **Dozzle:** agent v10.4.1 pinned to the hub's version. It needed an empty
+  `traefik-net` network.
+
+**Found while wiring (same "rots on re-IP" class as the a3 alias):**
+- The Dozzle hub still listed ana-ml2 `10.250.50.54` and irv-ml1's retired
+  `10.100.79.3`. fv-ml1 and irv-ml1 logs had been missing since their moves. Fixed;
+  the hub now reports 7 clients.
+- nh3-dev's Beszel agent had been down since the 09-24 NH3 power recovery (bind of
+  the automounted `/mnt/smithy` failed at boot; docker never retries a failed
+  create). Started; the Beszel "up" alert reached althing, an end-to-end proof of
+  the alert route. Recurs on the next cold start until fixed.
+- The nh3-docker Dozzle agent has been stopped by hand for ~5 months. Left alone.
+
+⚠ While checking the Beszel credential's shape, its password was printed into this
+session's transcript. The transcript stays on nh3-dev and it was not written to any
+memory; rotation offered to Prime.
@@ -117,13 +117,22 @@ no longer deployed sidecars here. See Recent decisions.)

 _As of 2026-09-24 ~2245 PT._

-### ⭐ TEI is the fleet embed/rerank engine; esh-ml1 is the SOLE backend (2026-09-25)
+### ⭐ esh-ml1 = the fleet's utility GPU host (2026-09-25): embed/rerank (TEI) + reward (vLLM)

-Cut-over done 0830 PT: gateway `qwen3-embedding` + `reranker` (+ DB alias a3) →
-TEI on esh-ml1; fv-ml1's two vLLM seats retired. ⚠ **Single backend until the
-second RTX 2000 arrives**, and **esh-ml1 has no monitoring yet**: wire Beszel +
-Kuma next. Next conversation with Prime: "what else moves" (vllm-reward is the
-candidate; **parakeet stays**). → Recent decisions.
+TEI is the fleet embed/rerank engine (Prime) and esh-ml1 is the SOLE backend
+(⚠ no failover until the second RTX 2000 arrives). The Skywork reward seat moved
+there too; the audit found nothing supersedes it. fv-ml1 GPU 1 freed ~15 GB
+across the day; only `vllm-coder` is left in `stacks/vllm`, and **parakeet stays**
+on fv-ml1. Monitoring is wired: Beszel with GPU and 5 alerts, Kuma #27/#28,
+Homepage cards, a Dozzle agent. Open follow-ups:
+- **worldtree-dev**: Domari's Skywork path is broken (dead IP + wrong schema).
+  Told 2026-09-25; waiting on them.
+- **nh3-dev Beszel agent** dies on every NH3 cold start (it binds an automounted
+  NAS share). Fix = drop its NAS extra-filesystems. Not done.
+- **Dozzle nh3-docker agent** has been stopped by hand since ~2026-04. Revive or drop.
+- **Beszel superuser password was echoed into this session's transcript**
+  (local only, not in memory). Rotation offered to Prime.
+→ Recent decisions.

 ### ✅ esh-ml1 built (2026-09-24) — two follow-ups

@@ -169,6 +178,7 @@ a shutdown (stays off by design). Outcomes and revert in

 ## Recent decisions

+- `[2026-09-25]` **Reward seat audited (Skywork-Reward-V2-Llama-3.1-8B still #1 of 188 on RewardBench 2; our AWQ ≈ bf16 within noise; double BOS costs ~2.7 pts) and moved fv-ml1 → esh-ml1; esh-ml1 monitoring wired (Beszel+GPU, Kuma, Homepage, Dozzle); Dozzle hub's stale agent IPs fixed; nh3-dev Beszel agent revived.** → `persistent-memory.d/2026-09-25-reward-move-and-esh-ml1-monitoring.md`
 - `[2026-09-25]` **TEI adopted as the fleet embed/rerank engine (Prime); esh-ml1 is the sole gateway backend for `qwen3-embedding` + `reranker`; fv-ml1's vLLM embed/rerank seats retired (~6.1 GB freed); parakeet stays on fv-ml1.** → `persistent-memory.d/2026-09-25-tei-fleet-embed-rerank.md`
 - `[2026-09-24]` **esh-ml1 built — RTX 2000E Ada as CT 110 on esh-pve (host driver 580.178.04 DKMS, loaded live, no reboot), vLLM embed+rerank at measured parity with fv-ml1, wired as LiteLLM `order: 2` failover; dead DB alias `reranker-a3-bge-v2-m3` (pre-relocation IP) repaired.** → `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md`
 - `[2026-09-24]` **esh-pve: VM 102 retired, a 14-day hung VFIO process found and cleared, T400 → RTX 2000 Ada; Prime chose LXC + host driver for embed/rerank** (implementation deferred to next session, tracked here + `servers/esh-pve/README.md`). → `persistent-memory.d/2026-09-24-esh-pve-vm102-retired-gpu-swap.md`
@@ -32,6 +32,7 @@ vars:
  ctid: 110
  hostname: esh-ml1
  ip_cidr: 10.0.50.80/24
+  ip_addr: 10.0.50.80
  gateway: 10.0.50.1
  vlan: 50
  cores: 6
@@ -156,6 +157,27 @@ steps:
      EOF
    when: "! pct exec {{ ctid }} -- sh -c 'grep -q nvidia /etc/docker/daemon.json && grep -Eq \"^no-cgroups *= *true\" /etc/nvidia-container-runtime/config.toml' 2>/dev/null"

+  # Fleet norm: Homepage (on esh-docker-vm) discovers labelled containers by
+  # reading every host's Docker API on tcp/2375 (stacks/homepage/conf/docker.yaml).
+  # Same unauthenticated plaintext exposure as fv-ml1 and esh-docker-vm, bound to
+  # this CT's one address. ⚠ Restarting dockerd restarts every container here,
+  # the fleet embed/rerank service included (~5 s for TEI, ~60 s for the reward seat).
+  - name: Expose the Docker API on tcp/2375 for Homepage discovery
+    shell: |
+      pct exec {{ ctid }} -- bash -s <<'EOF'
+      set -euo pipefail
+      install -d /etc/systemd/system/docker.service.d
+      cat > /etc/systemd/system/docker.service.d/override.conf <<'EOC'
+      # Homepage discovery — see eshpfi playbooks/esh-ml1-lxc.yaml
+      [Service]
+      ExecStart=
+      ExecStart=/usr/bin/dockerd -H fd:// -H tcp://{{ ip_addr }}:2375 --containerd=/run/containerd/containerd.sock
+      EOC
+      systemctl daemon-reload
+      systemctl restart docker
+      EOF
+    when: "! pct exec {{ ctid }} -- grep -q 'tcp://' /etc/systemd/system/docker.service.d/override.conf 2>/dev/null"
+
 verify:
  - name: Container is running with onboot set, started after the core guests
    shell: "pct status {{ ctid }} | grep -q running && pct config {{ ctid }} | grep -q '^onboot: 1' && pct config {{ ctid }} | grep -q '^startup: order=30'"
@@ -136,17 +136,20 @@ for each new kernel). Moving esh-pve to a different kernel series (6.14 opt-in)
 needs that series' headers meta-package installed first, or the module will not
 build and this CT will fail to start at the next boot.

-## Not yet wired
+## Monitoring and telemetry (wired 2026-09-25)

-⚠ **esh-ml1 became load-bearing on 2026-09-25, so these are no longer
-optional.** Nothing alerts if it dies today. The gateway will just start
-returning errors for `qwen3-embedding` and `reranker`.
+| layer | what | where |
+|---|---|---|
+| **Beszel** (host + GPU telemetry) | NVIDIA agent `henrygd/beszel-agent-nvidia:0.18.7`, `stacks/beszel` + `hosts/esh-ml1.yaml`. Hub system `ridkfdpwfq3f730`. GPU util, VRAM, power and temperature are sampled. | alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m (GPU included) |
+| **Uptime Kuma** (service) | `Embed — Qwen3 0.6B (TEI, esh-ml1)` → `:8001/health` (#27); `Rerank — bge-v2-m3 (TEI, esh-ml1)` → `:8013/health` (#28). TEI's health runs the backend. | alerts → infra-ops via althing-alert-bridge; `stacks/uptimekuma/monitors.yaml` |
+| **Homepage** | three cards under *AI - Eval & Retrieval*. dockerd exposes tcp/2375 on `10.0.50.80` only (fleet norm, `playbooks/esh-ml1-lxc.yaml`). | `stacks/homepage/conf/docker.yaml` → `esh-ml1-docker` |
+| **Dozzle** (logs) | agent `v10.4.1` on `10.0.50.80:7007`, compose dir `dozzle-agent` | hub on ana-docker :8088 |

- **Beszel**: no agent yet.
- **Uptime Kuma**: no check on `:8001/health` / `:8013/health` yet.
- **Homepage**: the compose carries labels, but esh-ml1 is not in
-  `stacks/homepage/conf/docker.yaml` (it would need dockerd on tcp/2375 like
-  the other hosts).
- **Path:** every consumer reaches esh-ml1 through the gateway at ana-docker, so
-  ESH callers hairpin ESH → Anaheim → ESH over the mesh. A mesh outage cuts
-  every consumer off, the ones at ESH included.
+The reward seat has **no Kuma check on purpose**: seats are outside Kuma's lane,
+and this one has no working consumer. Beszel and Homepage cover it.
+
+⚠ Disk: rootfs is ~66% used (vLLM image ~30 GB, TEI ~8 GB). The alert fires at 85%.
+
+**Path:** every consumer reaches esh-ml1 through the gateway at ana-docker, so
+ESH callers hairpin ESH → Anaheim → ESH over the mesh. A mesh outage cuts every
+consumer off, the ones at ESH included.
@@ -23,6 +23,16 @@ filesystem samples verified; fleet 13/14 up with known fv-ml1 outage.
 | irv-ml1 | beszel-agent-irv | /worktank, /storetank, /mnt/smithy |
 | vm-esh-nas | beszel-agent-esh-nas | /mnt/books, /mnt/share, /mnt/music, /mnt/media |
 | nh3-dev | beszel | /mnt/backup, /mnt/smithy |
+| esh-ml1 (added 2026-09-25) | beszel | none — NVIDIA image (`hosts/esh-ml1.yaml`) for the RTX 2000E Ada |
+
+⚠ **nh3-dev's agent was DOWN from the 2026-09-24 NH3 power recovery until
+2026-09-25.** Docker could not bind `/mnt/smithy` at boot ("no such device"): since
+`1cbde50` the NAS shares are automounted, and nh3-nas was not up yet. Docker does not
+retry a container that fails to *create*, so `unless-stopped` never brought it back.
+Started by hand. **It will recur on the next NH3 cold start** until the agent stops
+depending on NAS mounts at boot. The NAS capacity is already reported by
+nh3-nas's own agent, so dropping the two extra filesystems here is the simplest fix.
+Open item.

 Use `infra-ops@<ip>` with passwordless sudo, except vm-esh-nas:
 `lkraven@10.0.50.154` has Docker access. Irvine's hub address is
@@ -0,0 +1,14 @@
+# esh-ml1 (CT 110 on esh-pve) — Beszel agent with NVIDIA GPU telemetry for the
+# RTX 2000E Ada (utilization, VRAM, temperature, power). Docker-in-LXC: the
+# NVIDIA container toolkit runs with no-cgroups=true (playbooks/esh-ml1-lxc.yaml).
+# No extra filesystems: the root filesystem holds everything, models included.
+services:
+  beszel-agent:
+    image: henrygd/beszel-agent-nvidia:0.18.7
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [utility]
@@ -3,9 +3,19 @@
 Container log viewer. One UI on **ana-docker** aggregates logs from every Docker host via remote agents.

 **Deploys to:**
- **ana-docker** (hub) — UI at `http://10.250.50.70:8088`
+- **ana-docker** (hub) — UI at `http://10.250.50.70:8088`; live compose dir `/opt/docker/compose/dozzle-hub`
 - **fv-ml1** (agent) — listens on `10.251.50.54:7007`
- **nh3-docker** (agent, cross-site) — listens on `10.100.50.40:7007`
+- **esh-docker-vm** / **vm-esh-nas** (agents) — `10.0.50.45:7007`, `10.0.50.154:7007`
+- **irv-ml1** (agent) — `10.6.110.50:7007` (mesh address)
+- **esh-ml1** (agent, added 2026-09-25) — `10.0.50.80:7007`, pinned `v10.4.1` = the hub's version; compose dir `dozzle-agent`. The host needed an (empty) `traefik-net` network because the compose declares it external.
+- **nh3-docker** (agent, cross-site) — `10.100.50.40:7007`. ⚠ **Stopped by hand ~2026-04 (Exited 0) and left that way**; the hub logs a refused connection for it. Revive it or drop it from the list deliberately.
+
+⚠ **The hub's agent list silently rots when a host moves.** Until 2026-09-25 it
+still named ana-ml2's `10.250.50.54` and irv-ml1's retired `10.100.79.3`, so fv-ml1
+and irv-ml1 logs had been missing from Dozzle since their moves (2026-09-12 and
+2026-09-06). Nothing reported it. After any re-IP, check the hub's
+`DOZZLE_REMOTE_AGENT`, and check that `docker logs dozzle` shows `"clients":N` equal to
+reachable agents + 1.
 - **corviduo-dev** (agent) — listens on `10.250.50.152:7007`. Compose at `/home/vh/docker/compose/dozzle-agent/` (not `/opt/docker/compose/` — see `servers/corviduo-dev/README.md` for why)

 One compose.yaml lives on each host. The per-host `.env` sets `COMPOSE_PROFILES=hub` or `COMPOSE_PROFILES=agent` so `docker compose up -d` brings up the right service. On the hub, add every agent to `DOZZLE_REMOTE_AGENT` as a comma-separated list (e.g. `10.251.50.54:7007,10.100.50.40:7007`).
@@ -31,6 +31,12 @@ irv-ml1-docker:
  host: 10.6.110.50
  port: 2375

+# esh-ml1 — CT 110 on esh-pve: the fleet embed/rerank (TEI) + reward seats.
+# dockerd listens only on its own address (playbooks/esh-ml1-lxc.yaml), 2026-09-25.
+esh-ml1-docker:
+  host: 10.0.50.80
+  port: 2375
+
 # Example TLS socket (if/when a host moves off plaintext 2375):
 # ana-pfi-docker:
 #   host: 10.250.50.70
@@ -92,6 +92,21 @@ monitors:
  - name: Asset Engine
    url: http://10.250.50.70:8200/api/v1/services

+  # ---- fleet embed/rerank: the one EXCEPTION to "seats are OUT" ----
+  # Since 2026-09-25 these are the SOLE backends behind the gateway's
+  # `qwen3-embedding` and `reranker` (TEI on esh-ml1, no failover until the second
+  # RTX 2000 arrives). They are not come-and-go seats: dead = Worldtree recall,
+  # nevermore clustering and Open WebUI RAG all fail. TEI's /health runs the
+  # backend, so a loaded-but-broken model reads DOWN, not UP. The reward seat on the
+  # same box stays OUT: it has no working consumer (stacks/reward-seat/README.md).
+  - name: Embed — Qwen3 0.6B (TEI, esh-ml1)
+    url: http://10.0.50.80:8001/health
+    description: sole backend for gateway `qwen3-embedding`
+
+  - name: Rerank — bge-v2-m3 (TEI, esh-ml1)
+    url: http://10.0.50.80:8013/health
+    description: sole backend for gateway `reranker`
+
  - name: talk
    url: https://talk.nh3.phasefinal.com:8092/
    description: fleet voice bench