feat(coder-seat): Qwen2.5-Coder-1.5B copy on nh3-ml1 (not yet in the gateway)

Same model revision (df3ce67, blob sha matches fv-ml1), vLLM v0.24.0 and flags; 0.33 of the Ada (~5.4 GB, fv-ml1's absolute budget). Healthy, KV 79,808 tokens. Parity (teacher-forced true code, 40 FIM prompts, 2,215 tokens, cache-salted): each host is bit-exact with itself; cross-host |dlogprob| median 0.049, top-1 agreement 0.966 (Blackwell vs Ada + fp8 KV); against ground truth no quality difference (mean logprob diff +0.008 +/- 0.019 SE). Speed on-box: ~5x slower (64-tok FIM p50 ~1.0 s vs ~0.2 s; 63 vs 338 tok/s). Gateway coder-fast left on fv-ml1 pending Prime's call.
2026-09-25 23:56:34 -07:00
parent c923727cf2
commit 7f066a4b79
4 changed files with 164 additions and 0 deletions
@@ -33,6 +33,13 @@ Prime's call (recommendation: load-share; see below).

 VRAM ~2.7 GB for both, so ~13 GB is free.

+**Also here since 2026-09-25: a copy of the code-completion seat.**
+`stacks/coder-seat`, `vllm-coder` (Qwen2.5-Coder-1.5B, vLLM v0.24.0) on `:8020`,
+about 5 GB. It is **not behind the gateway yet**: on this card it runs about 5×
+slower than on fv-ml1 (a 64-token FIM completion takes ~1.0 s vs ~0.2 s), so
+Prime is deciding. The GPU total with TEI is ~7.6 of 16 GB. Details:
+`stacks/coder-seat/README.md`.
+
 ## Parity and speed vs esh-ml1 (2026-09-25 ~1540 PT)

 Client on nh3-dev, calling both seats directly. Corpus: 1,120 paragraphs from
@@ -0,0 +1,23 @@
+# coder-seat tunables (nh3-ml1). Copy to `.env` on the server.
+
+# Same vLLM as fv-ml1's seat, so the move changes only the GPU.
+VLLM_VERSION=v0.24.0
+
+# Same port as on fv-ml1, so the gateway entry changes only its host.
+CODER_PORT=8020
+
+CODER_MODEL=Qwen/Qwen2.5-Coder-1.5B
+CODER_REVISION=df3ce67c0e24480f20468b6ef2894622d69eb73b
+CODER_SERVED_NAME=qwen2.5-coder-1.5b
+CODER_MAX_MODEL_LEN=16384
+CODER_MAX_NUM_SEQS=32
+CODER_KV_CACHE_DTYPE=fp8
+
+# Fraction of the Ada's TOTAL 16,380 MiB: 0.33 ≈ 5.4 GB, the same absolute
+# budget fv-ml1 gave it (0.055 of 97,887 MiB). Weights ~3.1 GB bf16; fp8 KV is
+# ~14 KB/token, so the rest holds far more than one 16k sequence. vLLM checks
+# it against FREE memory at start; TEI holds ~2.6 GB of the card.
+CODER_GPU_MEM_UTIL=0.33
+
+# Matches fv-ml1: no key (internal network; the gateway fronts it).
+API_KEY=
@@ -0,0 +1,63 @@
+# coder-seat
+
+The fleet's small code-completion model on **nh3-ml1** (CT 109 on nh3-pve, RTX
+2000E Ada): **Qwen/Qwen2.5-Coder-1.5B** (the BASE model, used for
+fill-in-the-middle), vLLM v0.24.0, `:8020`. It shares the card with TEI
+embed/rerank (`stacks/embed-rerank`).
+
+Moved from fv-ml1's `vllm` stack on 2026-09-25 (Prime: utility seats off the
+Blackwells). It was chosen for Zed edit-predictions by a deep-research pass on
+2026-07-27 (Apache-2.0).
+
+| | |
+|---|---|
+| endpoint | `http://10.100.50.80:8020/v1/completions` (FIM: `<\|fim_prefix\|>…<\|fim_suffix\|>…<\|fim_middle\|>`) |
+| served name | `qwen2.5-coder-1.5b` |
+| gateway | `coder-fast` → `hosted_vllm/qwen2.5-coder-1.5b` (ana-docker:4000) |
+| revision | `df3ce67c0e24480f20468b6ef2894622d69eb73b` (pinned; same as fv-ml1 and HF main on 2026-09-25) |
+| memory | 0.33 of 16,380 MiB ≈ 5.4 GB, the same absolute budget as on fv-ml1 |
+
+## ⏸ Status 2026-09-25 2355: running, NOT behind the gateway (Prime deciding)
+
+The copy is up and parity-checked, but the gateway still points `coder-fast` at
+fv-ml1. On the 50 W Ada the same model is **about 5× slower**, which Zed
+edit-predictions would notice.
+
+**Speed, measured on each box** (3 interleaved reps each; fv-ml1 GPU 1 read 0%
+before and after):
+
+| workload | nh3-ml1 | fv-ml1 |
+|---|---|---|
+| time to first token (1 token), p50 | 20.3–20.9 ms | 7.9–9.7 ms |
+| FIM completion (1,500-char prefix, 64 tokens), p50 | **1,020–1,089 ms** | 198–206 ms |
+| decode, tokens/s | 62.7–63.2 | 337.6–337.7 |
+
+**Parity: same quality, different kernels.** The test uses 40 FIM prompts from
+this repo's scripts, with the file's real next 200 characters teacher-forced
+(2,215 tokens). Every run is a fresh prefill (`cache_salt`).
+- Each host repeats itself bit-exactly: the floor is |Δlogprob| 0 and top-1
+  agreement 1.0 on both.
+- Across hosts, per-token |Δlogprob| has median 0.049 and p99 1.72, and top-1
+  agreement is 0.966. The cause is Blackwell vs Ada kernels plus the fp8 KV
+  cache. The positive control (suffix dropped) gives median 1.46 and 0.58.
+- **Against the real code there is no difference.** Mean logprob is −1.6095
+  (nh3) vs −1.6178 (fv), a paired difference of +0.008 ± 0.019 SE. The true-token
+  top-1 rate is 0.739 vs 0.742 (−0.002 ± 0.003).
+- ⚠ Greedy exact-match is not a usable parity test for this seat. Same-host
+  greedy repeats matched only 30–38%, because run B hit the prefix cache and
+  took a numerically different path, and near-tie tokens cascade.
+
+## Consumers
+
+Everything goes through the gateway alias `coder-fast`. **Zed edit-predictions**
+reach it through `stacks/zed-fim-proxy` on ana-docker. The proxy injects a
+`coder-fast`-scoped virtual key and forwards to LiteLLM `:4000`, so only the
+gateway entry holds this seat's address. It saw 20 requests in the 7 days before
+the move.
+
+## Deploy
+
+```bash
+scripts/deploy-stack.sh nh3-ml1 coder-seat
+ssh nh3-ml1 'cd /opt/docker/compose/coder-seat && cp -n .env.example .env && docker compose config -q && docker compose up -d'
+```
@@ -0,0 +1,71 @@
+# coder-seat — the fleet's small code-completion model, on nh3-ml1 (CT 109 on
+# nh3-pve, RTX 2000E Ada, 16 GB).
+#
+# Moved here from fv-ml1's `vllm` stack on 2026-09-25 (Prime: utility seats off
+# the Blackwells). Same model revision, same vLLM version and flags; only the
+# GPU, the host and the memory fraction changed (the fraction is of a 16 GB card
+# now, sized to the same ~5.4 GB absolute budget).
+#
+#   vllm-coder   Qwen/Qwen2.5-Coder-1.5B (BASE, FIM) → /v1/completions   :8020
+#
+# Consumers: the gateway alias `coder-fast` (ana-docker:4000,
+# stacks/litellm/conf/config.yaml) and Zed edit-predictions (see README for how
+# Zed reaches it).
+#
+# Shares the card with TEI embed/rerank (stacks/embed-rerank, ~2.6 GB).
+name: coder-seat
+services:
+  vllm-coder:
+    image: vllm/vllm-openai:${VLLM_VERSION}
+    container_name: vllm-coder
+    restart: unless-stopped
+    ipc: host
+    ports:
+      - "${CODER_PORT}:8000"
+    volumes:
+      - /opt/aimodels/huggingface:/hfcache
+    environment:
+      - HF_HOME=/hfcache
+      - HF_HUB_CACHE=/hfcache/hub
+      - VLLM_API_KEY=${API_KEY:-}
+    command:
+      - ${CODER_MODEL}
+      # Pinned: the revision fv-ml1 served (and HF main as of 2026-09-25).
+      - --revision
+      - ${CODER_REVISION}
+      - --served-model-name
+      - ${CODER_SERVED_NAME}
+      - --host
+      - 0.0.0.0
+      - --port
+      - "8000"
+      - --gpu-memory-utilization
+      - ${CODER_GPU_MEM_UTIL}
+      - --max-model-len
+      - ${CODER_MAX_MODEL_LEN}
+      - --max-num-seqs
+      - ${CODER_MAX_NUM_SEQS}
+      - --dtype
+      - auto
+      - --kv-cache-dtype
+      - ${CODER_KV_CACHE_DTYPE}
+      - --enable-prefix-caching
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              device_ids: ["0"]
+              capabilities: [gpu]
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 300s
+    labels:
+      - homepage.group=AI - Inference
+      - homepage.name=vLLM Qwen2.5-Coder 1.5B (FIM, nh3-ml1)
+      - homepage.icon=mdi-code-braces
+      - homepage.description=Qwen2.5-Coder-1.5B FIM code-completion (gateway coder-fast, Zed edit-predictions)
+      - homepage.href=http://10.100.50.80:${CODER_PORT}/docs