vh
7f066a4b79
Same model revision (df3ce67, blob sha matches fv-ml1), vLLM v0.24.0 and flags; 0.33 of the Ada (~5.4 GB, fv-ml1's absolute budget). Healthy, KV 79,808 tokens. Parity (teacher-forced true code, 40 FIM prompts, 2,215 tokens, cache-salted): each host is bit-exact with itself; cross-host |dlogprob| median 0.049, top-1 agreement 0.966 (Blackwell vs Ada + fp8 KV); against ground truth no quality difference (mean logprob diff +0.008 +/- 0.019 SE). Speed on-box: ~5x slower (64-tok FIM p50 ~1.0 s vs ~0.2 s; 63 vs 338 tok/s). Gateway coder-fast left on fv-ml1 pending Prime's call.
24 lines
842 B
Bash
24 lines
842 B
Bash
# coder-seat tunables (nh3-ml1). Copy to `.env` on the server.
|
|
|
|
# Same vLLM as fv-ml1's seat, so the move changes only the GPU.
|
|
VLLM_VERSION=v0.24.0
|
|
|
|
# Same port as on fv-ml1, so the gateway entry changes only its host.
|
|
CODER_PORT=8020
|
|
|
|
CODER_MODEL=Qwen/Qwen2.5-Coder-1.5B
|
|
CODER_REVISION=df3ce67c0e24480f20468b6ef2894622d69eb73b
|
|
CODER_SERVED_NAME=qwen2.5-coder-1.5b
|
|
CODER_MAX_MODEL_LEN=16384
|
|
CODER_MAX_NUM_SEQS=32
|
|
CODER_KV_CACHE_DTYPE=fp8
|
|
|
|
# Fraction of the Ada's TOTAL 16,380 MiB: 0.33 ≈ 5.4 GB, the same absolute
|
|
# budget fv-ml1 gave it (0.055 of 97,887 MiB). Weights ~3.1 GB bf16; fp8 KV is
|
|
# ~14 KB/token, so the rest holds far more than one 16k sequence. vLLM checks
|
|
# it against FREE memory at start; TEI holds ~2.6 GB of the card.
|
|
CODER_GPU_MEM_UTIL=0.33
|
|
|
|
# Matches fv-ml1: no key (internal network; the gateway fronts it).
|
|
API_KEY=
|