vh
a300cdcd26
Deep-research-picked Qwen2.5-Coder-1.5B (BASE, Apache-2.0, native FIM) as a low-latency inline-completion seat: - stacks/vllm: vllm-coder service (ana-ml2 GPU1 :8020) + granite shrunk (util 0.27->0.13, max-len 131072->16384, seqs 1024->256; granite phasing out) to free GPU1 room. - stacks/litellm: coder-fast alias -> :8020 (mode: completion, /v1/completions). - stacks/zed-fim-proxy (NEW): keyless /v1/completions front door on ana-docker :4141 for Zed (which can't send an auth header) — POST + path + model allowlist, injects a coder-fast-scoped virtual key -> LiteLLM :4000. Anon /ping liveness. Verified keyless FIM end-to-end. Zed api_url = http://10.250.50.70:4141/v1, model coder-fast, prompt_format qwen. Source-IP allowlist off pending the Mac's observed source IP.
48 lines
2.4 KiB
Markdown
48 lines
2.4 KiB
Markdown
# zed-fim-proxy
|
||
|
||
A **keyless `/v1/completions` front door** on ana-docker for Zed's editor
|
||
edit-prediction (inline FIM completion) feature, which **cannot send an
|
||
`Authorization` header**. Runs on a separate port from LiteLLM and forwards to it
|
||
with an injected, model-scoped key.
|
||
|
||
- **Host:** ana-docker `10.250.50.70`, port **4141** (`network_mode: host`).
|
||
- **Backs:** the `coder-fast` model (Qwen2.5-Coder-1.5B FIM seat, `stacks/vllm`
|
||
`vllm-coder` on ana-ml2:8020) via LiteLLM `:4000`.
|
||
- **Zed config** (`edit_predictions.open_ai_compatible_api`): `api_url:
|
||
http://10.250.50.70:4141/v1`, `model: coder-fast`, `prompt_format: qwen`,
|
||
`max_output_tokens: <n>`. (The proxy also accepts `http://10.250.50.70:4141` —
|
||
it matches both `/v1/completions` and `/completions`.)
|
||
|
||
## Security model (stdlib proxy in `conf/proxy.py`)
|
||
|
||
Four guards + a scoped key — a keyless route that injects a working key is only
|
||
safe if it can't be pivoted:
|
||
|
||
1. **POST + path** `/v1/completions` (or `/completions`) only. `GET /ping` is an
|
||
anonymous liveness (`{"service":"ok"}`). `/v1/chat/completions` is rejected.
|
||
2. **Model allowlist** — request body `model` must equal `ZED_ALLOWED_MODEL`
|
||
(`coder-fast`); anything else → 403.
|
||
3. **Injected scoped key** — a LiteLLM virtual key scoped to `coder-fast` ONLY
|
||
(`POST /key/generate {"models":["coder-fast"]}`). Even if guards 1–2 were
|
||
bypassed, the key reaches nothing else (verified: 403 on `gen`). **This is the
|
||
real blast-radius bound.**
|
||
4. **Best-effort source-IP allowlist** (`ZED_ALLOWED_IPS`) — only enforceable if
|
||
the proxy sees the real client IP (hence `network_mode: host`; docker
|
||
port-publish would NAT it away). A site-to-site NAT may still mask the Mac's
|
||
`10.0.10.83` — verify against `docker logs zed-fim-proxy` and tighten. Internal
|
||
network only; no public exposure.
|
||
|
||
## Deploy
|
||
|
||
```
|
||
# conf/proxy.py -> /opt/docker/conf/zed-fim-proxy/proxy.py
|
||
# compose.yaml -> /opt/docker/compose/zed-fim-proxy/compose.yaml
|
||
# .env (from .env.example, with ZED_SCOPED_KEY filled) -> same dir, mode 600
|
||
cd /opt/docker/compose/zed-fim-proxy && docker compose up -d
|
||
# verify keyless:
|
||
curl -s http://localhost:4141/v1/completions -H 'Content-Type: application/json' \
|
||
-d '{"model":"coder-fast","prompt":"def add(a,b):\n return","max_tokens":16,"temperature":0.2}'
|
||
```
|
||
|
||
Stdlib-only proxy (no pip) in a bare `python:3.12-slim` container — no build.
|