esh-pfi-infrastructure/persistent-memory.d/2026-09-25-reward-move-and-esh-ml1-monitoring.md

[2026-09-25] Reward seat audit + move to esh-ml1; esh-ml1 monitoring wired

Prime: "Audit the reward seat to see if something has superseded what we deployed — then move it to esh-ml1 as well, then wire up telemetry and monitoring."

Audit (full record stacks/reward-seat/README.md).

• Not superseded. Skywork-Reward-V2-Llama-3.1-8B is #1 of 188 in AllenAI's official RewardBench 2 per-sample results (mean over the 5 non-Ties subsets, 84.7; the dataset was last updated 2025-12-11). No Skywork V3 exists. The -40M sibling is vendor-marked experimental. Caveat: 2026 models never submitted to RewardBench 2 would not appear.
• Our local AWQ W4A16: 0.847 vs published bf16 0.860 on 150 prompts (±2.9 pt SE), 96.2% pairwise agreement, Pearson 0.991.
• ⚠ vLLM adds a SECOND BOS to already-templated text unless add_special_tokens:false is sent: −2.7 pts. That was a harness confound in the first measurement (0.820), caught by checking /tokenize.
• No working consumer. 0 requests 2026-09-13 → 09-25. Worldtree Domari hardcodes the dead 10.250.50.54:8003 and sends a non-vLLM schema (prompt/responses → rewards), so it always fell back to Selene/chat-judge. worldtree-dev told (single message; their call).

Move.

• sha256-identical model copy.
• vLLM v0.24.0 on esh-ml1 :8003, util 0.55 (KV 21,328 tokens = 1.30× of a 16k request).
• Parity vs fv-ml1 on 600 RewardBench 2 texts:
  • 149/150 verdicts and 99.8% pairwise signs identical.
  • Raw |Δ| median 0.049; p99 0.22 vs the self-noise p99 of 0.16–0.20.
• Gateway /scalar-judge → 10.0.50.80:8003 (verified 3/3).
• fv-ml1 vllm-reward removed; stacks/vllm now = vllm-coder only.

Monitoring.

• Beszel: NVIDIA agent (GPU util/VRAM/power/temp), system ridkfdpwfq3f730, 5 alerts incl. Temperature >85 °C.
• Kuma: #27 embed, #28 rerank (the only seats inside Kuma's lane, because they are the sole backends). Reward excluded (no consumer).
• Homepage: dockerd tcp/2375 on 10.0.50.80, which restarted docker (all services healthy within ~31 s), and esh-ml1-docker in docker.yaml. Three cards appeared without a Homepage restart.
• Dozzle: agent v10.4.1 pinned to the hub's version. It needed an empty traefik-net network.

Found while wiring (same "rots on re-IP" class as the a3 alias):

• The Dozzle hub still listed ana-ml2 10.250.50.54 and irv-ml1's retired 10.100.79.3. fv-ml1 and irv-ml1 logs had been missing since their moves. Fixed; the hub now reports 7 clients.
• nh3-dev's Beszel agent had been down since the 09-24 NH3 power recovery (bind of the automounted /mnt/smithy failed at boot; docker never retries a failed create). Started; the Beszel "up" alert reached althing, an end-to-end proof of the alert route. Recurs on the next cold start until fixed.
• The nh3-docker Dozzle agent has been stopped by hand for ~5 months. Left alone.

⚠ While checking the Beszel credential's shape, its password was printed into this session's transcript. The transcript stays on nh3-dev and it was not written to any memory; rotation offered to Prime.