vh
7f066a4b79
Same model revision (df3ce67, blob sha matches fv-ml1), vLLM v0.24.0 and flags; 0.33 of the Ada (~5.4 GB, fv-ml1's absolute budget). Healthy, KV 79,808 tokens. Parity (teacher-forced true code, 40 FIM prompts, 2,215 tokens, cache-salted): each host is bit-exact with itself; cross-host |dlogprob| median 0.049, top-1 agreement 0.966 (Blackwell vs Ada + fp8 KV); against ground truth no quality difference (mean logprob diff +0.008 +/- 0.019 SE). Speed on-box: ~5x slower (64-tok FIM p50 ~1.0 s vs ~0.2 s; 63 vs 338 tok/s). Gateway coder-fast left on fv-ml1 pending Prime's call.
3.0 KiB
coder-seat
The fleet's small code-completion model on nh3-ml1 (CT 109 on nh3-pve, RTX
2000E Ada): Qwen/Qwen2.5-Coder-1.5B (the BASE model, used for
fill-in-the-middle), vLLM v0.24.0, :8020. It shares the card with TEI
embed/rerank (stacks/embed-rerank).
Moved from fv-ml1's vllm stack on 2026-09-25 (Prime: utility seats off the
Blackwells). It was chosen for Zed edit-predictions by a deep-research pass on
2026-07-27 (Apache-2.0).
| endpoint | http://10.100.50.80:8020/v1/completions (FIM: <|fim_prefix|>…<|fim_suffix|>…<|fim_middle|>) |
| served name | qwen2.5-coder-1.5b |
| gateway | coder-fast → hosted_vllm/qwen2.5-coder-1.5b (ana-docker:4000) |
| revision | df3ce67c0e24480f20468b6ef2894622d69eb73b (pinned; same as fv-ml1 and HF main on 2026-09-25) |
| memory | 0.33 of 16,380 MiB ≈ 5.4 GB, the same absolute budget as on fv-ml1 |
⏸ Status 2026-09-25 2355: running, NOT behind the gateway (Prime deciding)
The copy is up and parity-checked, but the gateway still points coder-fast at
fv-ml1. On the 50 W Ada the same model is about 5× slower, which Zed
edit-predictions would notice.
Speed, measured on each box (3 interleaved reps each; fv-ml1 GPU 1 read 0% before and after):
| workload | nh3-ml1 | fv-ml1 |
|---|---|---|
| time to first token (1 token), p50 | 20.3–20.9 ms | 7.9–9.7 ms |
| FIM completion (1,500-char prefix, 64 tokens), p50 | 1,020–1,089 ms | 198–206 ms |
| decode, tokens/s | 62.7–63.2 | 337.6–337.7 |
Parity: same quality, different kernels. The test uses 40 FIM prompts from
this repo's scripts, with the file's real next 200 characters teacher-forced
(2,215 tokens). Every run is a fresh prefill (cache_salt).
- Each host repeats itself bit-exactly: the floor is |Δlogprob| 0 and top-1 agreement 1.0 on both.
- Across hosts, per-token |Δlogprob| has median 0.049 and p99 1.72, and top-1 agreement is 0.966. The cause is Blackwell vs Ada kernels plus the fp8 KV cache. The positive control (suffix dropped) gives median 1.46 and 0.58.
- Against the real code there is no difference. Mean logprob is −1.6095 (nh3) vs −1.6178 (fv), a paired difference of +0.008 ± 0.019 SE. The true-token top-1 rate is 0.739 vs 0.742 (−0.002 ± 0.003).
- ⚠ Greedy exact-match is not a usable parity test for this seat. Same-host greedy repeats matched only 30–38%, because run B hit the prefix cache and took a numerically different path, and near-tie tokens cascade.
Consumers
Everything goes through the gateway alias coder-fast. Zed edit-predictions
reach it through stacks/zed-fim-proxy on ana-docker. The proxy injects a
coder-fast-scoped virtual key and forwards to LiteLLM :4000, so only the
gateway entry holds this seat's address. It saw 20 requests in the 7 days before
the move.
Deploy
scripts/deploy-stack.sh nh3-ml1 coder-seat
ssh nh3-ml1 'cd /opt/docker/compose/coder-seat && cp -n .env.example .env && docker compose config -q && docker compose up -d'