feat(esh-ml1): wire telemetry and monitoring; fix Dozzle's stale agent list

- Beszel: NVIDIA agent (stacks/beszel hosts/esh-ml1.yaml), hub system
  registered with Status/Disk/CPU/Memory/Temperature alerts; GPU samples
  verified (RTX 2000E Ada util, VRAM, power, temp).
- Uptime Kuma: health monitors for the TEI embed (:8001) and rerank (:8013)
  services — the sole backends behind the gateway, so the one exception to
  "seats are out of Kuma's lane". Reward seat excluded (no consumer).
- Homepage: dockerd on tcp/2375 bound to 10.0.50.80 (playbook step);
  esh-ml1-docker added to docker.yaml.
- Dozzle: agent v10.4.1 on esh-ml1. The hub's DOZZLE_REMOTE_AGENT still
  named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3; repointed
  to 10.251.50.54 / 10.6.110.50 (fv-ml1 and irv-ml1 logs were missing).
- Documented nh3-dev's Beszel agent failing on NH3 cold start (bind of an
  automounted NAS share); revived by hand, fix still open.
This commit is contained in:
vh
2026-09-25 09:20:36 -07:00
parent 52612cbe96
commit 65dc586497
9 changed files with 165 additions and 20 deletions
@@ -0,0 +1,55 @@
# `[2026-09-25]` Reward seat audit + move to esh-ml1; esh-ml1 monitoring wired
**Prime:** "Audit the reward seat to see if something has superseded what we
deployed — then move it to esh-ml1 as well, then wire up telemetry and monitoring."
**Audit (full record `stacks/reward-seat/README.md`).**
- Not superseded. Skywork-Reward-V2-Llama-3.1-8B is #1 of 188 in AllenAI's official
RewardBench 2 per-sample results (mean over the 5 non-Ties subsets, 84.7; the
dataset was last updated 2025-12-11). No Skywork V3 exists. The -40M sibling is
vendor-marked experimental. Caveat: 2026 models never submitted to RewardBench 2
would not appear.
- Our local AWQ W4A16: 0.847 vs published bf16 0.860 on 150 prompts (±2.9 pt SE),
96.2% pairwise agreement, Pearson 0.991.
- ⚠ vLLM adds a SECOND BOS to already-templated text unless
`add_special_tokens:false` is sent: −2.7 pts. That was a harness confound in the
first measurement (0.820), caught by checking `/tokenize`.
- **No working consumer.** 0 requests 2026-09-13 → 09-25. Worldtree Domari
hardcodes the dead `10.250.50.54:8003` and sends a non-vLLM schema
(`prompt`/`responses` → `rewards`), so it always fell back to Selene/chat-judge.
worldtree-dev told (single message; their call).
**Move.**
- sha256-identical model copy.
- vLLM v0.24.0 on esh-ml1 `:8003`, util 0.55 (KV 21,328 tokens = 1.30× of a 16k
request).
- Parity vs fv-ml1 on 600 RewardBench 2 texts:
- 149/150 verdicts and 99.8% pairwise signs identical.
- Raw |Δ| median 0.049; p99 0.22 vs the self-noise p99 of 0.16–0.20.
- Gateway `/scalar-judge` → `10.0.50.80:8003` (verified 3/3).
- fv-ml1 `vllm-reward` removed; `stacks/vllm` now = `vllm-coder` only.
**Monitoring.**
- **Beszel:** NVIDIA agent (GPU util/VRAM/power/temp), system
`ridkfdpwfq3f730`, 5 alerts incl. Temperature >85 °C.
- **Kuma:** #27 embed, #28 rerank (the only seats inside Kuma's lane, because they
are the sole backends). Reward excluded (no consumer).
- **Homepage:** dockerd tcp/2375 on 10.0.50.80, which restarted docker (all
services healthy within ~31 s), and `esh-ml1-docker` in docker.yaml. Three cards
appeared without a Homepage restart.
- **Dozzle:** agent v10.4.1 pinned to the hub's version. It needed an empty
`traefik-net` network.
**Found while wiring (same "rots on re-IP" class as the a3 alias):**
- The Dozzle hub still listed ana-ml2 `10.250.50.54` and irv-ml1's retired
`10.100.79.3`. fv-ml1 and irv-ml1 logs had been missing since their moves. Fixed;
the hub now reports 7 clients.
- nh3-dev's Beszel agent had been down since the 09-24 NH3 power recovery (bind of
the automounted `/mnt/smithy` failed at boot; docker never retries a failed
create). Started; the Beszel "up" alert reached althing, an end-to-end proof of
the alert route. Recurs on the next cold start until fixed.
- The nh3-docker Dozzle agent has been stopped by hand for ~5 months. Left alone.
⚠ While checking the Beszel credential's shape, its password was printed into this
session's transcript. The transcript stays on nh3-dev and it was not written to any
memory; rotation offered to Prime.