# `[2026-09-25]` Reward seat audit + move to esh-ml1; esh-ml1 monitoring wired

**Prime:** "Audit the reward seat to see if something has superseded what we
deployed — then move it to esh-ml1 as well, then wire up telemetry and monitoring."

**Audit (full record `stacks/reward-seat/README.md`).**
- Not superseded. Skywork-Reward-V2-Llama-3.1-8B is #1 of 188 in AllenAI's official
  RewardBench 2 per-sample results (mean over the 5 non-Ties subsets, 84.7; the
  dataset was last updated 2025-12-11). No Skywork V3 exists. The -40M sibling is
  vendor-marked experimental. Caveat: 2026 models never submitted to RewardBench 2
  would not appear.
- Our local AWQ W4A16: 0.847 vs published bf16 0.860 on 150 prompts (±2.9 pt SE),
  96.2% pairwise agreement, Pearson 0.991.
- ⚠ vLLM adds a SECOND BOS to already-templated text unless
  `add_special_tokens:false` is sent: −2.7 pts. That was a harness confound in the
  first measurement (0.820), caught by checking `/tokenize`.
- **No working consumer.** 0 requests 2026-09-13 → 09-25. Worldtree Domari
  hardcodes the dead `10.250.50.54:8003` and sends a non-vLLM schema
  (`prompt`/`responses` → `rewards`), so it always fell back to Selene/chat-judge.
  worldtree-dev told (single message; their call).

**Move.**
- sha256-identical model copy.
- vLLM v0.24.0 on esh-ml1 `:8003`, util 0.55 (KV 21,328 tokens = 1.30× of a 16k
  request).
- Parity vs fv-ml1 on 600 RewardBench 2 texts:
  - 149/150 verdicts and 99.8% pairwise signs identical.
  - Raw |Δ| median 0.049; p99 0.22 vs the self-noise p99 of 0.16–0.20.
- Gateway `/scalar-judge` → `10.0.50.80:8003` (verified 3/3).
- fv-ml1 `vllm-reward` removed; `stacks/vllm` now = `vllm-coder` only.

**Monitoring.**
- **Beszel:** NVIDIA agent (GPU util/VRAM/power/temp), system
  `ridkfdpwfq3f730`, 5 alerts incl. Temperature >85 °C.
- **Kuma:** #27 embed, #28 rerank (the only seats inside Kuma's lane, because they
  are the sole backends). Reward excluded (no consumer).
- **Homepage:** dockerd tcp/2375 on 10.0.50.80, which restarted docker (all
  services healthy within ~31 s), and `esh-ml1-docker` in docker.yaml. Three cards
  appeared without a Homepage restart.
- **Dozzle:** agent v10.4.1 pinned to the hub's version. It needed an empty
  `traefik-net` network.

**Found while wiring (same "rots on re-IP" class as the a3 alias):**
- The Dozzle hub still listed ana-ml2 `10.250.50.54` and irv-ml1's retired
  `10.100.79.3`. fv-ml1 and irv-ml1 logs had been missing since their moves. Fixed;
  the hub now reports 7 clients.
- nh3-dev's Beszel agent had been down since the 09-24 NH3 power recovery (bind of
  the automounted `/mnt/smithy` failed at boot; docker never retries a failed
  create). Started; the Beszel "up" alert reached althing, an end-to-end proof of
  the alert route. Recurs on the next cold start until fixed.
- The nh3-docker Dozzle agent has been stopped by hand for ~5 months. Left alone.

⚠ While checking the Beszel credential's shape, its password was printed into this
session's transcript. The transcript stays on nh3-dev and it was not written to any
memory; rotation offered to Prime.