2 Commits

Include renames

Author	SHA1	Message	Date
vh	1e2a3a13b5	tune(vllm): GPU-1 rebalance — granite 131k ctx, qwen 65k ctx, ~3.5GB free ... Reclaimed Qwen3.5-9B's over-provisioned KV (20x conc @ 32k) and handed it to granite. granite: 51200->131072 ctx (305k-token pool, 2.33x worst-case; PagedAttention => ~2.2x more short-request concurrency from the bigger pool), util 0.36->0.35. qwen: 32768->65536 ctx (8.13x), util 0.40->0.35. Trio unchanged (chunked inputs, 8k plenty). Leaves ~3.7GB free on the shared card. Start-order matters (trim qwen first, then grow granite) — vLLM requires free>=util*total at startup.	2026-06-13 08:46:46 -07:00
vh	2e3dcc2d3d	feat(qwen35-vl): Qwen3.5-9B VL FP8 stack on ana-ml2 GPU1 + LiteLLM entry ... Qwen3.5-9B vision-language served FP8 on ana-ml2 GPU1 (co-located with granite + the embed/rerank/reward trio; GPU0 kept free for hot-loading large models), :8007, fronted by LiteLLM as qwen3.5-9b-fp8. Pinned to vllm/vllm-openai nightly@sha256:49211ab2 — :latest (v0.19.1) quantizes the VL vision tower under fp8 and garbles vision; the nightly correctly excludes it (LM stays FP8, vision tower BF16). util 0.40 (~38GB) on the shared card (vLLM needs free>=util*total here). Vision verified end-to-end through the gateway.	2026-06-13 02:39:33 -07:00