vh
52612cbe96
- Audit: Skywork-Reward-V2-Llama-3.1-8B is still #1 of 188 on AllenAI's RewardBench 2 per-sample results; no Skywork V3; the -40M sibling is vendor-marked experimental. Our AWQ W4A16 quant: 0.847 vs published bf16 0.860 on a 150-prompt sample (within +/-2.9 pt SE), 96.2% pairwise agreement. Double BOS from vLLM on pre-templated text costs a further ~2.7 pts; callers must send add_special_tokens=false. - No working consumer: 0 requests since 2026-09-13; Worldtree Domari points at a dead IP with a non-vLLM schema (reported to worldtree-dev). - Move: sha256-identical model copy; vLLM v0.24.0 on esh-ml1 :8003 at 0.55 util (KV 1.30x of a 16k request). Parity vs fv-ml1: 149/150 verdicts, 99.8% pairwise signs, raw |delta| median 0.049. - Gateway /scalar-judge passthrough -> 10.0.50.80:8003; fv-ml1 vllm-reward removed (~10.2 GB freed on GPU 1). stacks/vllm now holds only vllm-coder.
76 lines
2.6 KiB
YAML
76 lines
2.6 KiB
YAML
# reward-seat — the fleet's scalar reward model, on esh-ml1 (RTX 2000E Ada, 16 GB).
|
|
#
|
|
# Moved here from fv-ml1's `vllm` stack on 2026-09-25 (Prime: utility seats off
|
|
# the Blackwells). Same model files (sha256-identical copy of fv-ml1's local AWQ
|
|
# output), same vLLM version and flags. Only the GPU and the host changed.
|
|
#
|
|
# vllm-reward Skywork-Reward-V2-Llama-3.1-8B, AWQ W4A16 (llm-compressor,
|
|
# score head kept full precision) → /classify :8003
|
|
#
|
|
# vLLM, not TEI: TEI cannot serve a Llama sequence-classification head. That
|
|
# makes this the one vLLM seat on esh-ml1; embed/rerank are TEI
|
|
# (stacks/embed-rerank).
|
|
#
|
|
# Scores are SWAP-SENSITIVE: a different model or quant shifts the scale and
|
|
# every consumer threshold with it. /classify returns sigmoid probs by default;
|
|
# send "use_activation": false for the raw reward.
|
|
#
|
|
# Consumers reach it through the gateway passthrough `/scalar-judge` on
|
|
# ana-docker:4000 (stacks/litellm, pass_through_endpoints). Worldtree's Domari
|
|
# hardcodes an address instead (see README).
|
|
#
|
|
# Model files: /opt/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ (local quant,
|
|
# not on HF Hub; the canonical copy also lives on fv-ml1 /tank/aimodels/llm).
|
|
|
|
name: reward-seat
|
|
|
|
services:
|
|
vllm-reward:
|
|
image: vllm/vllm-openai:${VLLM_VERSION}
|
|
container_name: vllm-reward
|
|
restart: unless-stopped
|
|
ipc: host
|
|
ports:
|
|
- "${REWARD_PORT}:8000"
|
|
volumes:
|
|
- /opt/aimodels/llm:/local-models:ro
|
|
environment:
|
|
- VLLM_API_KEY=${API_KEY:-}
|
|
command:
|
|
- /local-models/Skywork-Reward-V2-Llama-3.1-8B-AWQ
|
|
- --served-model-name
|
|
- Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ
|
|
# config.json declares LlamaForSequenceClassification; the pooling
|
|
# runner serves it as a single-label classifier.
|
|
- --runner
|
|
- pooling
|
|
- --host
|
|
- 0.0.0.0
|
|
- --port
|
|
- "8000"
|
|
- --gpu-memory-utilization
|
|
- ${REWARD_GPU_MEM_UTIL}
|
|
- --max-model-len
|
|
- ${REWARD_MAX_MODEL_LEN}
|
|
- --dtype
|
|
- auto
|
|
deploy:
|
|
resources:
|
|
reservations:
|
|
devices:
|
|
- driver: nvidia
|
|
device_ids: ["0"]
|
|
capabilities: [gpu]
|
|
healthcheck:
|
|
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 3
|
|
start_period: 240s
|
|
labels:
|
|
- homepage.group=AI - Eval & Retrieval
|
|
- homepage.name=Reward — Skywork V2 8B (esh-ml1)
|
|
- homepage.icon=mdi-scale-balance
|
|
- homepage.description=Scalar reward classifier (/scalar-judge) via vLLM on esh-ml1
|
|
- homepage.href=http://10.0.50.80:${REWARD_PORT}/docs
|