esh-pfi-infrastructure/stacks/reward-seat

reward-seat

The fleet's scalar reward model on esh-ml1 (CT 110 on esh-pve, RTX 2000E Ada): Skywork-Reward-V2-Llama-3.1-8B, our own AWQ W4A16 quant, on vLLM v0.24.0, :8003. Reached through the gateway passthrough http://10.250.50.70:4000/scalar-judge/<route> (gateway key required). Moved from fv-ml1 on 2026-09-25 (Prime: utility seats off the Blackwells).

Audit, 2026-09-25 — has anything superseded it? No.

• Still the best model of its kind. In AllenAI's official RewardBench 2 per-sample results (188 models, last updated 2025-12-11), this model ranks #1 by mean accuracy over the five non-Ties subsets (84.7; our aggregation approximates the official metric). The rest of Skywork's own V2 family fills #2–#4. There is no Skywork-Reward-V3. The sibling Skywork-Reward-V2-Llama-3.1-8B-40M scores higher on paper, but Skywork calls it experimental (about a third of its training pairs were flipped) and recommends the plain 8B for production. No credible 2026 release beat it in the HF or leaderboard searches, but models never submitted to RewardBench 2 would not show up there.

• Our quant is close to the original. RewardBench 2 sample: 150 prompts, 30 per non-Ties subset, 600 texts. Reference = the published bf16 scores.

  • Accuracy: AWQ 0.847 vs bf16 0.860. The standard error at n=150 is ±2.9 pts, so the difference is not resolved.
  • Pairwise preference-sign agreement 96.2%; raw-score Pearson 0.991; mean shift −0.60.
  • Largest per-subset gap: Precise IF 0.67 vs 0.77 (n=30).
  • Part of the remaining difference may be harness, not quant: the published run used HF transformers.

• ⚠ Double BOS costs accuracy. RewardBench texts already start with <|begin_of_text|>, and vLLM's /classify adds another unless the request sets "add_special_tokens": false. With the double BOS, accuracy fell to 0.820 (−2.7 pts) and Pearson to 0.981. Callers sending chat-templated text must set that flag.

• It has no working consumer. fv-ml1's seat served 0 requests between its 2026-09-13 start and this move. The only code caller, Worldtree's Domari (tools/domari/score_pair_skywork.py), is broken twice over:

  • it hardcodes ana-ml2's dead address 10.250.50.54:8003;
  • it sends {"prompt", "responses"} and reads {"rewards"}, which is not vLLM's /classify schema (input → data[].probs). It could never have succeeded against this seat, and its docstring says it falls back to the Selene/chat-judge path on error.

  Reported to worldtree-dev 2026-09-25.

Move, 2026-09-25

• Model files: a sha256-identical copy of fv-ml1's /tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ (7 files, 4.4 GB), now at /opt/aimodels/llm/ on esh-ml1. fv-ml1 keeps the canonical copy.

• Same vLLM image digest and flags. --gpu-memory-utilization 0.55 (≈9.0 GB of 16,380 MiB) → KV 21,328 tokens = 1.30× a full 16,384-token request. GPU total with TEI: ~10.4 GB used.

• Parity vs fv-ml1 (the same 600 texts, 2 runs per site):

  	raw-score |Δ| median	p99	max
  fv-ml1 vs itself	0.000	0.198	0.297
  esh-ml1 vs itself	0.002	0.157	0.199
  esh-ml1 vs fv-ml1	0.049	0.221	0.321

  149/150 prompt verdicts are identical, and 99.8% of pairwise preference signs. Different GPUs add a small offset (median 0.05 against a score range of tens), but the tails match each site's own noise.

• Gateway /scalar-judge repointed to 10.0.50.80:8003; verified 3/3 through the gateway. fv-ml1's vllm-reward was removed, freeing ~10.2 GB on GPU 1.

Smoke test

curl -s http://10.0.50.80:8003/classify -H 'content-type: application/json' \
  -d '{"model":"Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ","input":"User: hello\nAssistant: hi there","use_activation":false}' \
  | jq '.data[0].probs[0]'        # raw reward, ~4.27

Options not taken

• FP8 (W8A8) quant, native on Ada, would likely track bf16 more closely at ~8.5 GB of weights (it fits beside TEI, tightly). Not pursued: the quant already sits within the measurement's resolution, nothing consumes the seat, and any quant swap shifts the score scale for consumers.