67 lines
3.3 KiB
Markdown
67 lines
3.3 KiB
Markdown
# coder-seat
|
||
|
||
The fleet's small code-completion model on **nh3-ml1** (CT 109 on nh3-pve, RTX
|
||
2000E Ada): **Qwen/Qwen2.5-Coder-1.5B** (the BASE model, used for
|
||
fill-in-the-middle), vLLM v0.24.0, `:8020`. It shares the card with TEI
|
||
embed/rerank (`stacks/embed-rerank`).
|
||
|
||
Moved from fv-ml1's `vllm` stack on 2026-09-25 (Prime: utility seats off the
|
||
Blackwells). It was chosen for Zed edit-predictions by a deep-research pass on
|
||
2026-07-27 (Apache-2.0).
|
||
|
||
| | |
|
||
|---|---|
|
||
| endpoint | `http://10.100.50.80:8020/v1/completions` (FIM: `<\|fim_prefix\|>…<\|fim_suffix\|>…<\|fim_middle\|>`) |
|
||
| served name | `qwen2.5-coder-1.5b` |
|
||
| gateway | `coder-fast` → `hosted_vllm/qwen2.5-coder-1.5b` (ana-docker:4000) |
|
||
| revision | `df3ce67c0e24480f20468b6ef2894622d69eb73b` (pinned; same as fv-ml1 and HF main on 2026-09-25) |
|
||
| memory | 0.33 of 16,380 MiB ≈ 5.4 GB, the same absolute budget as on fv-ml1 |
|
||
|
||
## ❌ NOT DEPLOYED. Decision (Prime, 2026-09-26 0000): the seat stays on fv-ml1
|
||
|
||
The nh3-ml1 copy was built, checked for parity, measured, and then removed (container,
|
||
image, model cache). It was the same quality but **about 5× slower** on the 50 W
|
||
Ada. That lag would reach Zed edit-predictions, and moving it would only have freed
|
||
6.3 GB on an fv-ml1 GPU that already had about 20 GB spare. The live seat is still
|
||
`stacks/vllm` on fv-ml1. This stack is kept as the tested recipe in case the
|
||
calculation changes (a faster card, or speculative decoding).
|
||
|
||
**Speed, measured on each box** (3 interleaved reps each; fv-ml1 GPU 1 read 0%
|
||
before and after):
|
||
|
||
| workload | nh3-ml1 | fv-ml1 |
|
||
|---|---|---|
|
||
| time to first token (1 token), p50 | 20.3–20.9 ms | 7.9–9.7 ms |
|
||
| FIM completion (1,500-char prefix, 64 tokens), p50 | **1,020–1,089 ms** | 198–206 ms |
|
||
| decode, tokens/s | 62.7–63.2 | 337.6–337.7 |
|
||
|
||
**Parity: same quality, different kernels.** The test uses 40 FIM prompts from
|
||
this repo's scripts, with the file's real next 200 characters teacher-forced
|
||
(2,215 tokens). Every run is a fresh prefill (`cache_salt`).
|
||
- Each host repeats itself bit-exactly: the floor is |Δlogprob| 0 and top-1
|
||
agreement 1.0 on both.
|
||
- Across hosts, per-token |Δlogprob| has median 0.049 and p99 1.72, and top-1
|
||
agreement is 0.966. The cause is Blackwell vs Ada kernels plus the fp8 KV
|
||
cache. The positive control (suffix dropped) gives median 1.46 and 0.58.
|
||
- **Against the real code there is no difference.** Mean logprob is −1.6095
|
||
(nh3) vs −1.6178 (fv), a paired difference of +0.008 ± 0.019 SE. The true-token
|
||
top-1 rate is 0.739 vs 0.742 (−0.002 ± 0.003).
|
||
- ⚠ Greedy exact-match is not a usable parity test for this seat. Same-host
|
||
greedy repeats matched only 30–38%, because run B hit the prefix cache and
|
||
took a numerically different path, and near-tie tokens cascade.
|
||
|
||
## Consumers
|
||
|
||
Everything goes through the gateway alias `coder-fast`. **Zed edit-predictions**
|
||
reach it through `stacks/zed-fim-proxy` on ana-docker. The proxy injects a
|
||
`coder-fast`-scoped virtual key and forwards to LiteLLM `:4000`, so only the
|
||
gateway entry holds this seat's address. It saw 20 requests in the 7 days before
|
||
the move.
|
||
|
||
## Deploy
|
||
|
||
```bash
|
||
scripts/deploy-stack.sh nh3-ml1 coder-seat
|
||
ssh nh3-ml1 'cd /opt/docker/compose/coder-seat && cp -n .env.example .env && docker compose config -q && docker compose up -d'
|
||
```
|