# reward-seat

The fleet's scalar reward model on **esh-ml1** (CT 110 on esh-pve, RTX 2000E
Ada): **Skywork-Reward-V2-Llama-3.1-8B**, our own AWQ W4A16 quant, on vLLM
v0.24.0, `:8003`. Reached through the gateway passthrough
`http://10.250.50.70:4000/scalar-judge/<route>` (gateway key required).
Moved from fv-ml1 on 2026-09-25 (Prime: utility seats off the Blackwells).

## Audit, 2026-09-25 — has anything superseded it? No.

- **Still the best model of its kind.** In AllenAI's official RewardBench 2
  per-sample results (188 models, last updated 2025-12-11), this model ranks
  **#1** by mean accuracy over the five non-Ties subsets (84.7; our aggregation
  approximates the official metric). The rest of Skywork's own V2 family fills
  #2–#4. There is **no Skywork-Reward-V3**. The sibling
  `Skywork-Reward-V2-Llama-3.1-8B-40M` scores higher on paper, but Skywork calls
  it *experimental* (about a third of its training pairs were flipped) and
  recommends the plain 8B for production. No credible 2026 release beat it in
  the HF or leaderboard searches, but models never submitted to RewardBench 2
  would not show up there.
- **Our quant is close to the original.** RewardBench 2 sample: 150 prompts, 30
  per non-Ties subset, 600 texts. Reference = the published bf16 scores.
  - Accuracy: AWQ **0.847** vs bf16 **0.860**. The standard error at n=150 is
    ±2.9 pts, so the difference is not resolved.
  - Pairwise preference-sign agreement 96.2%; raw-score Pearson 0.991; mean
    shift −0.60.
  - Largest per-subset gap: Precise IF 0.67 vs 0.77 (n=30).
  - Part of the remaining difference may be harness, not quant: the published
    run used HF transformers.
- ⚠ **Double BOS costs accuracy.** RewardBench texts already start with
  `<|begin_of_text|>`, and vLLM's `/classify` adds another unless the request
  sets `"add_special_tokens": false`. With the double BOS, accuracy fell to
  **0.820** (−2.7 pts) and Pearson to 0.981. Callers sending chat-templated
  text must set that flag.
- **It has no working consumer.** fv-ml1's seat served 0 requests between its
  2026-09-13 start and this move. The only code caller, Worldtree's Domari
  (`tools/domari/score_pair_skywork.py`), is broken twice over:
  - it hardcodes ana-ml2's dead address `10.250.50.54:8003`;
  - it sends `{"prompt", "responses"}` and reads `{"rewards"}`, which is not
    vLLM's `/classify` schema (`input` → `data[].probs`). It could never have
    succeeded against this seat, and its docstring says it falls back to the
    Selene/chat-judge path on error.

  Reported to worldtree-dev 2026-09-25.

## Move, 2026-09-25

- Model files: a sha256-identical copy of fv-ml1's
  `/tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ` (7 files, 4.4 GB), now at
  `/opt/aimodels/llm/` on esh-ml1. fv-ml1 keeps the canonical copy.
- Same vLLM image digest and flags. `--gpu-memory-utilization 0.55` (≈9.0 GB of
  16,380 MiB) → KV 21,328 tokens = 1.30× a full 16,384-token request. GPU total
  with TEI: ~10.4 GB used.
- **Parity vs fv-ml1** (the same 600 texts, 2 runs per site):

  | | raw-score \|Δ\| median | p99 | max |
  |---|---|---|---|
  | fv-ml1 vs itself | 0.000 | 0.198 | 0.297 |
  | esh-ml1 vs itself | 0.002 | 0.157 | 0.199 |
  | **esh-ml1 vs fv-ml1** | **0.049** | 0.221 | 0.321 |

  149/150 prompt verdicts are identical, and 99.8% of pairwise preference
  signs. Different GPUs add a small offset (median 0.05 against a score range
  of tens), but the tails match each site's own noise.
- Gateway `/scalar-judge` repointed to `10.0.50.80:8003`; verified 3/3 through
  the gateway. fv-ml1's `vllm-reward` was removed, freeing ~10.2 GB on GPU 1.

## Smoke test

```bash
curl -s http://10.0.50.80:8003/classify -H 'content-type: application/json' \
  -d '{"model":"Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ","input":"User: hello\nAssistant: hi there","use_activation":false}' \
  | jq '.data[0].probs[0]'        # raw reward, ~4.27
```

## Options not taken

- **FP8 (W8A8) quant**, native on Ada, would likely track bf16 more closely
  at ~8.5 GB of weights (it fits beside TEI, tightly). Not pursued: the quant
  already sits within the measurement's resolution, nothing consumes the seat,
  and any quant swap shifts the score scale for consumers.