vh
52612cbe96
- Audit: Skywork-Reward-V2-Llama-3.1-8B is still #1 of 188 on AllenAI's RewardBench 2 per-sample results; no Skywork V3; the -40M sibling is vendor-marked experimental. Our AWQ W4A16 quant: 0.847 vs published bf16 0.860 on a 150-prompt sample (within +/-2.9 pt SE), 96.2% pairwise agreement. Double BOS from vLLM on pre-templated text costs a further ~2.7 pts; callers must send add_special_tokens=false. - No working consumer: 0 requests since 2026-09-13; Worldtree Domari points at a dead IP with a non-vLLM schema (reported to worldtree-dev). - Move: sha256-identical model copy; vLLM v0.24.0 on esh-ml1 :8003 at 0.55 util (KV 1.30x of a 16k request). Parity vs fv-ml1: 149/150 verdicts, 99.8% pairwise signs, raw |delta| median 0.049. - Gateway /scalar-judge passthrough -> 10.0.50.80:8003; fv-ml1 vllm-reward removed (~10.2 GB freed on GPU 1). stacks/vllm now holds only vllm-coder.
82 lines
4.2 KiB
Markdown
82 lines
4.2 KiB
Markdown
# reward-seat
|
||
|
||
The fleet's scalar reward model on **esh-ml1** (CT 110 on esh-pve, RTX 2000E
|
||
Ada): **Skywork-Reward-V2-Llama-3.1-8B**, our own AWQ W4A16 quant, on vLLM
|
||
v0.24.0, `:8003`. Reached through the gateway passthrough
|
||
`http://10.250.50.70:4000/scalar-judge/<route>` (gateway key required).
|
||
Moved from fv-ml1 on 2026-09-25 (Prime: utility seats off the Blackwells).
|
||
|
||
## Audit, 2026-09-25 — has anything superseded it? No.
|
||
|
||
- **Still the best model of its kind.** In AllenAI's official RewardBench 2
|
||
per-sample results (188 models, last updated 2025-12-11), this model ranks
|
||
**#1** by mean accuracy over the five non-Ties subsets (84.7; our aggregation
|
||
approximates the official metric). The rest of Skywork's own V2 family fills
|
||
#2–#4. There is **no Skywork-Reward-V3**. The sibling
|
||
`Skywork-Reward-V2-Llama-3.1-8B-40M` scores higher on paper, but Skywork calls
|
||
it *experimental* (about a third of its training pairs were flipped) and
|
||
recommends the plain 8B for production. No credible 2026 release beat it in
|
||
the HF or leaderboard searches, but models never submitted to RewardBench 2
|
||
would not show up there.
|
||
- **Our quant is close to the original.** RewardBench 2 sample: 150 prompts, 30
|
||
per non-Ties subset, 600 texts. Reference = the published bf16 scores.
|
||
- Accuracy: AWQ **0.847** vs bf16 **0.860**. The standard error at n=150 is
|
||
±2.9 pts, so the difference is not resolved.
|
||
- Pairwise preference-sign agreement 96.2%; raw-score Pearson 0.991; mean
|
||
shift −0.60.
|
||
- Largest per-subset gap: Precise IF 0.67 vs 0.77 (n=30).
|
||
- Part of the remaining difference may be harness, not quant: the published
|
||
run used HF transformers.
|
||
- ⚠ **Double BOS costs accuracy.** RewardBench texts already start with
|
||
`<|begin_of_text|>`, and vLLM's `/classify` adds another unless the request
|
||
sets `"add_special_tokens": false`. With the double BOS, accuracy fell to
|
||
**0.820** (−2.7 pts) and Pearson to 0.981. Callers sending chat-templated
|
||
text must set that flag.
|
||
- **It has no working consumer.** fv-ml1's seat served 0 requests between its
|
||
2026-09-13 start and this move. The only code caller, Worldtree's Domari
|
||
(`tools/domari/score_pair_skywork.py`), is broken twice over:
|
||
- it hardcodes ana-ml2's dead address `10.250.50.54:8003`;
|
||
- it sends `{"prompt", "responses"}` and reads `{"rewards"}`, which is not
|
||
vLLM's `/classify` schema (`input` → `data[].probs`). It could never have
|
||
succeeded against this seat, and its docstring says it falls back to the
|
||
Selene/chat-judge path on error.
|
||
|
||
Reported to worldtree-dev 2026-09-25.
|
||
|
||
## Move, 2026-09-25
|
||
|
||
- Model files: a sha256-identical copy of fv-ml1's
|
||
`/tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ` (7 files, 4.4 GB), now at
|
||
`/opt/aimodels/llm/` on esh-ml1. fv-ml1 keeps the canonical copy.
|
||
- Same vLLM image digest and flags. `--gpu-memory-utilization 0.55` (≈9.0 GB of
|
||
16,380 MiB) → KV 21,328 tokens = 1.30× a full 16,384-token request. GPU total
|
||
with TEI: ~10.4 GB used.
|
||
- **Parity vs fv-ml1** (the same 600 texts, 2 runs per site):
|
||
|
||
| | raw-score \|Δ\| median | p99 | max |
|
||
|---|---|---|---|
|
||
| fv-ml1 vs itself | 0.000 | 0.198 | 0.297 |
|
||
| esh-ml1 vs itself | 0.002 | 0.157 | 0.199 |
|
||
| **esh-ml1 vs fv-ml1** | **0.049** | 0.221 | 0.321 |
|
||
|
||
149/150 prompt verdicts are identical, and 99.8% of pairwise preference
|
||
signs. Different GPUs add a small offset (median 0.05 against a score range
|
||
of tens), but the tails match each site's own noise.
|
||
- Gateway `/scalar-judge` repointed to `10.0.50.80:8003`; verified 3/3 through
|
||
the gateway. fv-ml1's `vllm-reward` was removed, freeing ~10.2 GB on GPU 1.
|
||
|
||
## Smoke test
|
||
|
||
```bash
|
||
curl -s http://10.0.50.80:8003/classify -H 'content-type: application/json' \
|
||
-d '{"model":"Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ","input":"User: hello\nAssistant: hi there","use_activation":false}' \
|
||
| jq '.data[0].probs[0]' # raw reward, ~4.27
|
||
```
|
||
|
||
## Options not taken
|
||
|
||
- **FP8 (W8A8) quant**, native on Ada, would likely track bf16 more closely
|
||
at ~8.5 GB of weights (it fits beside TEI, tightly). Not pursued: the quant
|
||
already sits within the measurement's resolution, nothing consumes the seat,
|
||
and any quant swap shifts the score scale for consumers.
|