vh
52612cbe96
- Audit: Skywork-Reward-V2-Llama-3.1-8B is still #1 of 188 on AllenAI's RewardBench 2 per-sample results; no Skywork V3; the -40M sibling is vendor-marked experimental. Our AWQ W4A16 quant: 0.847 vs published bf16 0.860 on a 150-prompt sample (within +/-2.9 pt SE), 96.2% pairwise agreement. Double BOS from vLLM on pre-templated text costs a further ~2.7 pts; callers must send add_special_tokens=false. - No working consumer: 0 requests since 2026-09-13; Worldtree Domari points at a dead IP with a non-vLLM schema (reported to worldtree-dev). - Move: sha256-identical model copy; vLLM v0.24.0 on esh-ml1 :8003 at 0.55 util (KV 1.30x of a 16k request). Parity vs fv-ml1: 149/150 verdicts, 99.8% pairwise signs, raw |delta| median 0.049. - Gateway /scalar-judge passthrough -> 10.0.50.80:8003; fv-ml1 vllm-reward removed (~10.2 GB freed on GPU 1). stacks/vllm now holds only vllm-coder.
19 lines
746 B
Bash
19 lines
746 B
Bash
# reward-seat tunables (esh-ml1). Copy to `.env` on the server.
|
|
|
|
# Same vLLM as fv-ml1's seat had, so the move changed only the GPU. The image
|
|
# is already on esh-ml1 from the vLLM embed/rerank era.
|
|
VLLM_VERSION=v0.24.0
|
|
|
|
# Same port as on fv-ml1, so the gateway passthrough changed only its host.
|
|
REWARD_PORT=8003
|
|
|
|
# Fraction of the Ada's TOTAL 16,380 MiB (≈9.0 GB). vLLM checks it against
|
|
# free memory at start; TEI embed+rerank hold ~2.6 GB of the rest. The KV pool
|
|
# must hold one full 16,384-token sequence (~2.1 GB for Llama-3.1-8B in bf16).
|
|
REWARD_GPU_MEM_UTIL=0.55
|
|
# Server-side cap; Worldtree's Domari budgets 14,000 tokens against it.
|
|
REWARD_MAX_MODEL_LEN=16384
|
|
|
|
# Matches fv-ml1: no key (the gateway passthrough fronts it).
|
|
API_KEY=
|