vh
65dc586497
- Beszel: NVIDIA agent (stacks/beszel hosts/esh-ml1.yaml), hub system registered with Status/Disk/CPU/Memory/Temperature alerts; GPU samples verified (RTX 2000E Ada util, VRAM, power, temp). - Uptime Kuma: health monitors for the TEI embed (:8001) and rerank (:8013) services — the sole backends behind the gateway, so the one exception to "seats are out of Kuma's lane". Reward seat excluded (no consumer). - Homepage: dockerd on tcp/2375 bound to 10.0.50.80 (playbook step); esh-ml1-docker added to docker.yaml. - Dozzle: agent v10.4.1 on esh-ml1. The hub's DOZZLE_REMOTE_AGENT still named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3; repointed to 10.251.50.54 / 10.6.110.50 (fv-ml1 and irv-ml1 logs were missing). - Documented nh3-dev's Beszel agent failing on NH3 cold start (bind of an automounted NAS share); revived by hand, fix still open.
56 lines
3.1 KiB
Markdown
56 lines
3.1 KiB
Markdown
# `[2026-09-25]` Reward seat audit + move to esh-ml1; esh-ml1 monitoring wired
|
||
|
||
**Prime:** "Audit the reward seat to see if something has superseded what we
|
||
deployed — then move it to esh-ml1 as well, then wire up telemetry and monitoring."
|
||
|
||
**Audit (full record `stacks/reward-seat/README.md`).**
|
||
- Not superseded. Skywork-Reward-V2-Llama-3.1-8B is #1 of 188 in AllenAI's official
|
||
RewardBench 2 per-sample results (mean over the 5 non-Ties subsets, 84.7; the
|
||
dataset was last updated 2025-12-11). No Skywork V3 exists. The -40M sibling is
|
||
vendor-marked experimental. Caveat: 2026 models never submitted to RewardBench 2
|
||
would not appear.
|
||
- Our local AWQ W4A16: 0.847 vs published bf16 0.860 on 150 prompts (±2.9 pt SE),
|
||
96.2% pairwise agreement, Pearson 0.991.
|
||
- ⚠ vLLM adds a SECOND BOS to already-templated text unless
|
||
`add_special_tokens:false` is sent: −2.7 pts. That was a harness confound in the
|
||
first measurement (0.820), caught by checking `/tokenize`.
|
||
- **No working consumer.** 0 requests 2026-09-13 → 09-25. Worldtree Domari
|
||
hardcodes the dead `10.250.50.54:8003` and sends a non-vLLM schema
|
||
(`prompt`/`responses` → `rewards`), so it always fell back to Selene/chat-judge.
|
||
worldtree-dev told (single message; their call).
|
||
|
||
**Move.**
|
||
- sha256-identical model copy.
|
||
- vLLM v0.24.0 on esh-ml1 `:8003`, util 0.55 (KV 21,328 tokens = 1.30× of a 16k
|
||
request).
|
||
- Parity vs fv-ml1 on 600 RewardBench 2 texts:
|
||
- 149/150 verdicts and 99.8% pairwise signs identical.
|
||
- Raw |Δ| median 0.049; p99 0.22 vs the self-noise p99 of 0.16–0.20.
|
||
- Gateway `/scalar-judge` → `10.0.50.80:8003` (verified 3/3).
|
||
- fv-ml1 `vllm-reward` removed; `stacks/vllm` now = `vllm-coder` only.
|
||
|
||
**Monitoring.**
|
||
- **Beszel:** NVIDIA agent (GPU util/VRAM/power/temp), system
|
||
`ridkfdpwfq3f730`, 5 alerts incl. Temperature >85 °C.
|
||
- **Kuma:** #27 embed, #28 rerank (the only seats inside Kuma's lane, because they
|
||
are the sole backends). Reward excluded (no consumer).
|
||
- **Homepage:** dockerd tcp/2375 on 10.0.50.80, which restarted docker (all
|
||
services healthy within ~31 s), and `esh-ml1-docker` in docker.yaml. Three cards
|
||
appeared without a Homepage restart.
|
||
- **Dozzle:** agent v10.4.1 pinned to the hub's version. It needed an empty
|
||
`traefik-net` network.
|
||
|
||
**Found while wiring (same "rots on re-IP" class as the a3 alias):**
|
||
- The Dozzle hub still listed ana-ml2 `10.250.50.54` and irv-ml1's retired
|
||
`10.100.79.3`. fv-ml1 and irv-ml1 logs had been missing since their moves. Fixed;
|
||
the hub now reports 7 clients.
|
||
- nh3-dev's Beszel agent had been down since the 09-24 NH3 power recovery (bind of
|
||
the automounted `/mnt/smithy` failed at boot; docker never retries a failed
|
||
create). Started; the Beszel "up" alert reached althing, an end-to-end proof of
|
||
the alert route. Recurs on the next cold start until fixed.
|
||
- The nh3-docker Dozzle agent has been stopped by hand for ~5 months. Left alone.
|
||
|
||
⚠ While checking the Beszel credential's shape, its password was printed into this
|
||
session's transcript. The transcript stays on nh3-dev and it was not written to any
|
||
memory; rotation offered to Prime.
|