feat(vllm): add phi4-mini FP8 summarizer/dreamer on ana-ml2; retire granite from llama-swap

phi4-mini supersedes the granite-4-small llama-swap pin as the summarizer + dreaming agent. New vllm-phi4 service: Phi-4-mini-instruct, vLLM-native FP8 (near-lossless on RTX 6000 Ada cc 8.9), 50K ctx, FP8 KV cache, GPU 1, :8004. llama-swap: removed granite-4-small (depinned) + granite-4-micro config — both superseded. CONFIG ONLY; the GGUFs stay on disk. Frees granite-4-small's ~24 GB (it was pinned at 120K ctx). Placement: phi4 on GPU 1 with the embed/rerank/reward trio (~1.2 GB margin at 50K); keeps GPU 0 clear for llama-swap heavy models. docs/roadmap.md captures the deferred vLLM observability (Langfuse req/resp tracing + Prometheus/Grafana). Deploy order: llama-swap config (free granite) -> vllm-phi4 -> repoint nevermore.
2026-06-03 23:21:45 -07:00
parent 5cd4678d9f
commit 40a374b809
4 changed files with 151 additions and 37 deletions
@@ -197,6 +197,70 @@ services:
      - homepage.description=Skywork-Reward-V2 8B classifier via vLLM (ana-ml2)
      - homepage.href=http://10.250.50.54:${REWARD_PORT}/docs

+  # Phi-4-mini (FP8) — summarizer + "dreaming" agent. Supersedes the
+  # llama-swap granite-4-small pin. Generative chat model (OpenAI
+  # /v1/chat/completions), so NO --runner pooling. FP8 on RTX 6000 Ada
+  # (cc 8.9): near-lossless, ~1.2x, ~6 GB.
+  vllm-phi4:
+    image: vllm/vllm-openai:${VLLM_VERSION}
+    container_name: vllm-phi4
+    restart: unless-stopped
+    ipc: host
+    ports:
+      - "${PHI4_PORT}:8000"
+    volumes:
+      - /tank/aimodels/huggingface:/hfcache
+    environment:
+      - HF_HOME=/hfcache
+      - HF_HUB_CACHE=/hfcache/hub
+      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
+      - VLLM_API_KEY=${API_KEY:-}
+    command:
+      # Default = vLLM-native dynamic FP8 from the official BF16 weights (most
+      # robust on Ada). Alt = pre-quantized checkpoint: PHI4_MODEL=
+      # pytorch/Phi-4-mini-instruct-FP8 + PHI4_QUANT=torchao (see .env/README).
+      - ${PHI4_MODEL}
+      - --served-model-name
+      - ${PHI4_SERVED_NAME}
+      - --quantization
+      - ${PHI4_QUANT}
+      - --host
+      - 0.0.0.0
+      - --port
+      - "8000"
+      - --gpu-memory-utilization
+      - ${PHI4_GPU_MEM_UTIL}
+      - --max-model-len
+      - ${PHI4_MAX_MODEL_LEN}
+      - --dtype
+      - auto
+      # FP8 KV cache — halves KV memory at 128K ctx on Ada (cc 8.9); near-lossless.
+      - --kv-cache-dtype
+      - ${PHI4_KV_CACHE_DTYPE}
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              device_ids:
+                - "${PHI4_GPU_ID}"
+              capabilities:
+                - gpu
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 180s
+    networks:
+      - tnet
+    labels:
+      - homepage.group=AI Systems
+      - homepage.name=vLLM Phi-4-mini (summarizer/dreamer)
+      - homepage.icon=mdi-text-box-outline
+      - homepage.description=Phi-4-mini FP8 via vLLM (ana-ml2)
+      - homepage.href=http://10.250.50.54:${PHI4_PORT}/docs
+
 networks:
  tnet:
    name: traefik-net