From 7f066a4b79342e4bee611def498524638fc5e34a Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 25 Sep 2026 23:56:34 -0700 Subject: [PATCH] feat(coder-seat): Qwen2.5-Coder-1.5B copy on nh3-ml1 (not yet in the gateway) Same model revision (df3ce67, blob sha matches fv-ml1), vLLM v0.24.0 and flags; 0.33 of the Ada (~5.4 GB, fv-ml1's absolute budget). Healthy, KV 79,808 tokens. Parity (teacher-forced true code, 40 FIM prompts, 2,215 tokens, cache-salted): each host is bit-exact with itself; cross-host |dlogprob| median 0.049, top-1 agreement 0.966 (Blackwell vs Ada + fp8 KV); against ground truth no quality difference (mean logprob diff +0.008 +/- 0.019 SE). Speed on-box: ~5x slower (64-tok FIM p50 ~1.0 s vs ~0.2 s; 63 vs 338 tok/s). Gateway coder-fast left on fv-ml1 pending Prime's call. --- servers/nh3-ml1/README.md | 7 ++++ stacks/coder-seat/.env.example | 23 +++++++++++ stacks/coder-seat/README.md | 63 ++++++++++++++++++++++++++++++ stacks/coder-seat/compose.yaml | 71 ++++++++++++++++++++++++++++++++++ 4 files changed, 164 insertions(+) create mode 100644 stacks/coder-seat/.env.example create mode 100644 stacks/coder-seat/README.md create mode 100644 stacks/coder-seat/compose.yaml diff --git a/servers/nh3-ml1/README.md b/servers/nh3-ml1/README.md index e2f0af3..b8b971d 100644 --- a/servers/nh3-ml1/README.md +++ b/servers/nh3-ml1/README.md @@ -33,6 +33,13 @@ Prime's call (recommendation: load-share; see below). VRAM ~2.7 GB for both, so ~13 GB is free. +**Also here since 2026-09-25: a copy of the code-completion seat.** +`stacks/coder-seat`, `vllm-coder` (Qwen2.5-Coder-1.5B, vLLM v0.24.0) on `:8020`, +about 5 GB. It is **not behind the gateway yet**: on this card it runs about 5× +slower than on fv-ml1 (a 64-token FIM completion takes ~1.0 s vs ~0.2 s), so +Prime is deciding. The GPU total with TEI is ~7.6 of 16 GB. Details: +`stacks/coder-seat/README.md`. + ## Parity and speed vs esh-ml1 (2026-09-25 ~1540 PT) Client on nh3-dev, calling both seats directly. Corpus: 1,120 paragraphs from diff --git a/stacks/coder-seat/.env.example b/stacks/coder-seat/.env.example new file mode 100644 index 0000000..933be3c --- /dev/null +++ b/stacks/coder-seat/.env.example @@ -0,0 +1,23 @@ +# coder-seat tunables (nh3-ml1). Copy to `.env` on the server. + +# Same vLLM as fv-ml1's seat, so the move changes only the GPU. +VLLM_VERSION=v0.24.0 + +# Same port as on fv-ml1, so the gateway entry changes only its host. +CODER_PORT=8020 + +CODER_MODEL=Qwen/Qwen2.5-Coder-1.5B +CODER_REVISION=df3ce67c0e24480f20468b6ef2894622d69eb73b +CODER_SERVED_NAME=qwen2.5-coder-1.5b +CODER_MAX_MODEL_LEN=16384 +CODER_MAX_NUM_SEQS=32 +CODER_KV_CACHE_DTYPE=fp8 + +# Fraction of the Ada's TOTAL 16,380 MiB: 0.33 ≈ 5.4 GB, the same absolute +# budget fv-ml1 gave it (0.055 of 97,887 MiB). Weights ~3.1 GB bf16; fp8 KV is +# ~14 KB/token, so the rest holds far more than one 16k sequence. vLLM checks +# it against FREE memory at start; TEI holds ~2.6 GB of the card. +CODER_GPU_MEM_UTIL=0.33 + +# Matches fv-ml1: no key (internal network; the gateway fronts it). +API_KEY= diff --git a/stacks/coder-seat/README.md b/stacks/coder-seat/README.md new file mode 100644 index 0000000..463c587 --- /dev/null +++ b/stacks/coder-seat/README.md @@ -0,0 +1,63 @@ +# coder-seat + +The fleet's small code-completion model on **nh3-ml1** (CT 109 on nh3-pve, RTX +2000E Ada): **Qwen/Qwen2.5-Coder-1.5B** (the BASE model, used for +fill-in-the-middle), vLLM v0.24.0, `:8020`. It shares the card with TEI +embed/rerank (`stacks/embed-rerank`). + +Moved from fv-ml1's `vllm` stack on 2026-09-25 (Prime: utility seats off the +Blackwells). It was chosen for Zed edit-predictions by a deep-research pass on +2026-07-27 (Apache-2.0). + +| | | +|---|---| +| endpoint | `http://10.100.50.80:8020/v1/completions` (FIM: `<\|fim_prefix\|>…<\|fim_suffix\|>…<\|fim_middle\|>`) | +| served name | `qwen2.5-coder-1.5b` | +| gateway | `coder-fast` → `hosted_vllm/qwen2.5-coder-1.5b` (ana-docker:4000) | +| revision | `df3ce67c0e24480f20468b6ef2894622d69eb73b` (pinned; same as fv-ml1 and HF main on 2026-09-25) | +| memory | 0.33 of 16,380 MiB ≈ 5.4 GB, the same absolute budget as on fv-ml1 | + +## ⏸ Status 2026-09-25 2355: running, NOT behind the gateway (Prime deciding) + +The copy is up and parity-checked, but the gateway still points `coder-fast` at +fv-ml1. On the 50 W Ada the same model is **about 5× slower**, which Zed +edit-predictions would notice. + +**Speed, measured on each box** (3 interleaved reps each; fv-ml1 GPU 1 read 0% +before and after): + +| workload | nh3-ml1 | fv-ml1 | +|---|---|---| +| time to first token (1 token), p50 | 20.3–20.9 ms | 7.9–9.7 ms | +| FIM completion (1,500-char prefix, 64 tokens), p50 | **1,020–1,089 ms** | 198–206 ms | +| decode, tokens/s | 62.7–63.2 | 337.6–337.7 | + +**Parity: same quality, different kernels.** The test uses 40 FIM prompts from +this repo's scripts, with the file's real next 200 characters teacher-forced +(2,215 tokens). Every run is a fresh prefill (`cache_salt`). +- Each host repeats itself bit-exactly: the floor is |Δlogprob| 0 and top-1 + agreement 1.0 on both. +- Across hosts, per-token |Δlogprob| has median 0.049 and p99 1.72, and top-1 + agreement is 0.966. The cause is Blackwell vs Ada kernels plus the fp8 KV + cache. The positive control (suffix dropped) gives median 1.46 and 0.58. +- **Against the real code there is no difference.** Mean logprob is −1.6095 + (nh3) vs −1.6178 (fv), a paired difference of +0.008 ± 0.019 SE. The true-token + top-1 rate is 0.739 vs 0.742 (−0.002 ± 0.003). +- ⚠ Greedy exact-match is not a usable parity test for this seat. Same-host + greedy repeats matched only 30–38%, because run B hit the prefix cache and + took a numerically different path, and near-tie tokens cascade. + +## Consumers + +Everything goes through the gateway alias `coder-fast`. **Zed edit-predictions** +reach it through `stacks/zed-fim-proxy` on ana-docker. The proxy injects a +`coder-fast`-scoped virtual key and forwards to LiteLLM `:4000`, so only the +gateway entry holds this seat's address. It saw 20 requests in the 7 days before +the move. + +## Deploy + +```bash +scripts/deploy-stack.sh nh3-ml1 coder-seat +ssh nh3-ml1 'cd /opt/docker/compose/coder-seat && cp -n .env.example .env && docker compose config -q && docker compose up -d' +``` diff --git a/stacks/coder-seat/compose.yaml b/stacks/coder-seat/compose.yaml new file mode 100644 index 0000000..8e064a7 --- /dev/null +++ b/stacks/coder-seat/compose.yaml @@ -0,0 +1,71 @@ +# coder-seat — the fleet's small code-completion model, on nh3-ml1 (CT 109 on +# nh3-pve, RTX 2000E Ada, 16 GB). +# +# Moved here from fv-ml1's `vllm` stack on 2026-09-25 (Prime: utility seats off +# the Blackwells). Same model revision, same vLLM version and flags; only the +# GPU, the host and the memory fraction changed (the fraction is of a 16 GB card +# now, sized to the same ~5.4 GB absolute budget). +# +# vllm-coder Qwen/Qwen2.5-Coder-1.5B (BASE, FIM) → /v1/completions :8020 +# +# Consumers: the gateway alias `coder-fast` (ana-docker:4000, +# stacks/litellm/conf/config.yaml) and Zed edit-predictions (see README for how +# Zed reaches it). +# +# Shares the card with TEI embed/rerank (stacks/embed-rerank, ~2.6 GB). +name: coder-seat +services: + vllm-coder: + image: vllm/vllm-openai:${VLLM_VERSION} + container_name: vllm-coder + restart: unless-stopped + ipc: host + ports: + - "${CODER_PORT}:8000" + volumes: + - /opt/aimodels/huggingface:/hfcache + environment: + - HF_HOME=/hfcache + - HF_HUB_CACHE=/hfcache/hub + - VLLM_API_KEY=${API_KEY:-} + command: + - ${CODER_MODEL} + # Pinned: the revision fv-ml1 served (and HF main as of 2026-09-25). + - --revision + - ${CODER_REVISION} + - --served-model-name + - ${CODER_SERVED_NAME} + - --host + - 0.0.0.0 + - --port + - "8000" + - --gpu-memory-utilization + - ${CODER_GPU_MEM_UTIL} + - --max-model-len + - ${CODER_MAX_MODEL_LEN} + - --max-num-seqs + - ${CODER_MAX_NUM_SEQS} + - --dtype + - auto + - --kv-cache-dtype + - ${CODER_KV_CACHE_DTYPE} + - --enable-prefix-caching + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: ["0"] + capabilities: [gpu] + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8000/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 300s + labels: + - homepage.group=AI - Inference + - homepage.name=vLLM Qwen2.5-Coder 1.5B (FIM, nh3-ml1) + - homepage.icon=mdi-code-braces + - homepage.description=Qwen2.5-Coder-1.5B FIM code-completion (gateway coder-fast, Zed edit-predictions) + - homepage.href=http://10.100.50.80:${CODER_PORT}/docs