feat: Zed edit-predictions keyless FIM route (Qwen2.5-Coder-1.5B / coder-fast)
Deep-research-picked Qwen2.5-Coder-1.5B (BASE, Apache-2.0, native FIM) as a low-latency inline-completion seat: - stacks/vllm: vllm-coder service (ana-ml2 GPU1 :8020) + granite shrunk (util 0.27->0.13, max-len 131072->16384, seqs 1024->256; granite phasing out) to free GPU1 room. - stacks/litellm: coder-fast alias -> :8020 (mode: completion, /v1/completions). - stacks/zed-fim-proxy (NEW): keyless /v1/completions front door on ana-docker :4141 for Zed (which can't send an auth header) — POST + path + model allowlist, injects a coder-fast-scoped virtual key -> LiteLLM :4000. Anon /ping liveness. Verified keyless FIM end-to-end. Zed api_url = http://10.250.50.70:4141/v1, model coder-fast, prompt_format qwen. Source-IP allowlist off pending the Mac's observed source IP.
This commit is contained in:
@@ -278,6 +278,70 @@ services:
|
||||
- homepage.description=Granite 4.1 8B FP8 via vLLM (ana-ml2)
|
||||
- homepage.href=http://10.250.50.54:${GRANITE_PORT}/docs
|
||||
|
||||
vllm-coder:
|
||||
image: vllm/vllm-openai:${VLLM_VERSION}
|
||||
container_name: vllm-coder
|
||||
restart: unless-stopped
|
||||
ipc: host
|
||||
ports:
|
||||
- "${CODER_PORT}:8000"
|
||||
volumes:
|
||||
- /tank/aimodels/huggingface:/hfcache
|
||||
environment:
|
||||
- HF_HOME=/hfcache
|
||||
- HF_HUB_CACHE=/hfcache/hub
|
||||
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
|
||||
- VLLM_API_KEY=${API_KEY:-}
|
||||
command:
|
||||
# Qwen2.5-Coder-1.5B (BASE) — FIM code-completion seat for Zed edit-predictions
|
||||
# (deep-research pick 2026-07-27). Native fill-in-the-middle: <|fim_prefix|> /
|
||||
# <|fim_suffix|> / <|fim_middle|> (IDs 151659/151660/151661); Zed sends the
|
||||
# FIM-formatted prompt to /v1/completions and vLLM passes it through (the FIM
|
||||
# special tokens live in the tokenizer). BASE not -Instruct (FIM is a
|
||||
# pretraining objective; base completions are cleaner). Apache-2.0. Runner-up =
|
||||
# Qwen2.5-Coder-3B (higher HumanEval-FIM but non-commercial Qwen-Research license).
|
||||
- ${CODER_MODEL}
|
||||
- --served-model-name
|
||||
- ${CODER_SERVED_NAME}
|
||||
- --host
|
||||
- 0.0.0.0
|
||||
- --port
|
||||
- "8000"
|
||||
- --gpu-memory-utilization
|
||||
- ${CODER_GPU_MEM_UTIL}
|
||||
- --max-model-len
|
||||
- ${CODER_MAX_MODEL_LEN}
|
||||
- --max-num-seqs
|
||||
- ${CODER_MAX_NUM_SEQS}
|
||||
- --dtype
|
||||
- auto
|
||||
- --kv-cache-dtype
|
||||
- ${CODER_KV_CACHE_DTYPE}
|
||||
- --enable-prefix-caching
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids:
|
||||
- "${CODER_GPU_ID}"
|
||||
capabilities:
|
||||
- gpu
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 300s
|
||||
networks:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=AI - Inference
|
||||
- homepage.name=vLLM Qwen2.5-Coder 1.5B (FIM)
|
||||
- homepage.icon=mdi-code-braces
|
||||
- homepage.description=Qwen2.5-Coder-1.5B FIM code-completion (ana-ml2, Zed edit-predictions)
|
||||
- homepage.href=http://10.250.50.54:${CODER_PORT}/docs
|
||||
|
||||
networks:
|
||||
tnet:
|
||||
name: traefik-net
|
||||
|
||||
Reference in New Issue
Block a user