From 05f3d3b2a33673904246193ebcada09ffcbf04bb Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 14 Sep 2026 10:25:10 -0700 Subject: [PATCH] feat(fv-ml1): add gen-small A3B seat; sec/gen/char to native 262K; coder reclaim; gateway Full seat rebalance across GPU0/GPU1 (flash on GPU2 and empty GPU3 untouched), operator-directed. Every target seat now serves native 262,144 context with concurrency in the requested 1.2-2.5x band, verified from live boot logs: cyberprev (sec) 262144 @ 1.37x depth-probed CLEAN to 259,722 tokens flash-next (gen) 262144 @ 1.31x (untouched, already in band) gen-small (NEW) 262144 @ 2.56x MTP k=3 measured 69.6% accept / 3.09 len char-rp 262144 @ 1.22x (was 1.02x; util 0.40->0.52) char-rp-fast 262144 @ 2.04x (util cap 0.30->0.24, pinned KV unchanged) - gen-small: NEW seat, Qwen3.6-35B-A3B (3B active MoE), llmfan46 Heretic (MPOA) NVFP4 experts-only, already on disk at qwen36-35b-a3b-heretic-nvfp4. There is no general Qwen3.8 A3B (3.8 MoEs are Flash-Next and the 2.4T), so this is the 3.6 fallback the operator specified. GPU0, :8026, MTP k=3, coherent and MTP-verified before wiring. gen-small / gen-small-reasoning gateway aliases. - coder: 8192 @ 13.75x -> 16384 @ 4.70x (util 0.06->0.055). Context doubled, waste cut. Not the exact 2-3x target: the 1.5B weight+overhead floor (~4.2 GiB) sits just under the util knob's resolution, so hitting <=3x reliably needs a --kv-cache-memory byte pin (compose change) rather than the util fraction. - cyberprev raised 163840 -> 262144: depth-probed with non-repeating prompts to 259,722 tokens, clean (no OOM, memory flat). Unlike mog-sec (same base arch, capped at 163840 for depth crashes), this checkpoint holds native depth. - Gateway (operator calls): summarizer + classifier -> gen-small; new classifier-large -> gen-large (flash) for the accuracy tier; summarizer-large stays on flash. All verified end-to-end. - GPU1 hit its ceiling raising char-rp; resolved by trimming char-rp-fast's reservation cap (its KV is pinned, so concurrency held at 2.04x) rather than moving a utility seat -- the shared GPU_ID on reward/embed/rerank made a single-seat move messier than the in-GPU rebalance. Seat inventory regenerated from the live containers. --- docs/pfi/fv-ml1-gpu-seat-inventory.md | 31 +++++-- stacks/gen-small-seat/.env.example | 38 +++++++++ stacks/gen-small-seat/compose.yaml | 115 ++++++++++++++++++++++++++ stacks/litellm/conf/config.yaml | 76 +++++++++++++---- 4 files changed, 233 insertions(+), 27 deletions(-) create mode 100644 stacks/gen-small-seat/.env.example create mode 100644 stacks/gen-small-seat/compose.yaml diff --git a/docs/pfi/fv-ml1-gpu-seat-inventory.md b/docs/pfi/fv-ml1-gpu-seat-inventory.md index 444350d..f94f8fb 100644 --- a/docs/pfi/fv-ml1-gpu-seat-inventory.md +++ b/docs/pfi/fv-ml1-gpu-seat-inventory.md @@ -6,7 +6,7 @@ Hand-edits are overwritten and, worse, drift silently while looking authoritative — which is exactly the failure this file replaced. --> -**Generated 2026-09-14 16:10 UTC** by `scripts/seat-inventory.py`, read from the running +**Generated 2026-09-14 17:24 UTC** by `scripts/seat-inventory.py`, read from the running containers on `100.64.0.7` — `docker inspect`, `nvidia-smi`, each model's own `config.json`, and the `.PROVENANCE.txt` siblings on `/tank`. @@ -17,11 +17,12 @@ containers on `100.64.0.7` — `docker inspect`, `nvidia-smi`, each model's own | GPU | seat | VRAM | weights | KV tokens | ctx | concurrency | util | |---|---|---|---|---|---|---|---| -| 0 | `vllm-cyberprev` | 38.0 GiB | 25.11 GiB | 166,264 | 163840 | **1.01×** | 0.40 | -| 1 | `vllm-meromero-rp` | 37.9 GiB | 19.51 GiB | 266,334 | 262144 | **1.02×** | 0.40 | -| 1 | `vllm-erp-seat` | 26.3 GiB | 15.9 GiB | 534,649 | 262144 | **2.04×** | 0.30 | +| 0 | `vllm-cyberprev` | 46.0 GiB | 25.11 GiB | 359,372 | 262144 | **1.37×** | 0.40 | +| 0 | `vllm-gen-small` | 35.3 GiB | 23.98 GiB | 670,142 | 262144 | **2.56×** | 0.48 | +| 1 | `vllm-meromero-rp` | 41.3 GiB | 19.51 GiB | 320,774 | 262144 | **1.22×** | 0.52 | +| 1 | `vllm-erp-seat` | 26.4 GiB | 15.9 GiB | 534,649 | 262144 | **2.04×** | 0.24 | | 1 | `vllm-reward` | 9.0 GiB | 4.41 GiB | 26,224 | 16384 | **1.60×** | 0.10 | -| 1 | `vllm-coder` | 6.0 GiB | 2.98 GiB | 112,624 | 8192 | **13.75×** | 0.06 | +| 1 | `vllm-coder` | 5.5 GiB | 2.98 GiB | 77,056 | 16384 | **4.70×** | 0.055 | | 1 | `vllm-embed` | 3.4 GiB | 1.12 GiB | 10,272 | 8192 | **1.25×** | 0.03 | | 1 | `vllm-rerank-a3` | 2.1 GiB | 1.06 GiB | — | 8192 | — | 0.03 | | 2 | `vllm-flash-next` | 93.1 GiB | 76.82 GiB | 344,155 | 262144 | **1.31×** | 0.96 | @@ -41,11 +42,20 @@ context. - **speculative decoding:** `{"method": "dflash", "model": "/drafter", "num_speculative_tokens": 7}` - **image:** `vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013` +### `vllm-gen-small` — GPU 0 + +- **serves:** `gen-small`, `gen-small-thinking` +- **model:** `/tank/aimodels/qwen36-35b-a3b-heretic-nvfp4` +- **architecture:** Qwen3_5MoeForConditionalGeneration (qwen3_5_moe), 40 layers, 256 experts +- **quantization:** modelopt / None — W4A4 (None) +- **speculative decoding:** `{"method": "qwen3_5_mtp", "num_speculative_tokens": 3}` +- **image:** `vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013` + ### `vllm-coder` — GPU 1 - **serves:** `qwen2.5-coder-1.5b` - **model:** `?` -- **image:** `vllm/vllm-openai:latest` ⚠ **floating tag** +- **image:** `vllm/vllm-openai:v0.24.0` ### `vllm-embed` — GPU 1 @@ -94,7 +104,7 @@ context. ## Gateway aliases resolving to this host -19 aliases. Ports with no listening seat are marked dead. +22 aliases. Ports with no listening seat are marked dead. | alias | port | |---|---| @@ -102,20 +112,23 @@ context. | `char-rp-fast` | 8021 | | `char-rp-reasoning` | 8016 | | `chat-judge` | 8022 | -| `classifier` | 8022 | +| `classifier` | 8026 | +| `classifier-large` | 8022 | | `coder-fast` | 8020 | | `erp-tune-v2` | 8098 | | `gemma4-26b-a4b-it-base` | 8099 | | `gen` | 8022 | | `gen-large` | 8022 | | `gen-reasoning` | 8022 | +| `gen-small` | 8026 | +| `gen-small-reasoning` | 8026 | | `image-judge` | 8022 | | `qwen-image-bench` | 8022 | | `qwen3-embedding` | 8001 | | `reranker` | 8013 | | `sec` | 8025 | | `sec-reasoning` | 8025 | -| `summarizer` | 8022 | +| `summarizer` | 8026 | | `summarizer-large` | 8022 | --- diff --git a/stacks/gen-small-seat/.env.example b/stacks/gen-small-seat/.env.example new file mode 100644 index 0000000..2776090 --- /dev/null +++ b/stacks/gen-small-seat/.env.example @@ -0,0 +1,38 @@ +# gen-small-seat — Qwen3.6-35B-A3B Heretic (llmfan46), fv-ml1 GPU 0, :8026. +# Copy to .env on the host at /opt/docker/compose/gen-small-seat/.env. + +GEN_SMALL_IMAGE=vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013 +API_KEY= + +# ── Placement ── shares GPU 0 with cyberprev (sec). KV pinned in bytes so the two +# co-resident seats don't fight over a util fraction. +GEN_SMALL_GPU_ID=0 +GEN_SMALL_PORT=8026 +GEN_SMALL_CONTAINER_NAME=vllm-gen-small + +# ── Model ── llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-NVFP4-Experts-Only +# Heretic v1.3.0 (MPOA); modelopt NVFP4 experts-only; 19 MTP preserved. +GEN_SMALL_MODEL=/tank/aimodels/qwen36-35b-a3b-heretic-nvfp4 +GEN_SMALL_QUANT=modelopt_fp4 +GEN_SMALL_SERVED_NAME=gen-small +GEN_SMALL_SERVED_NAME_THINK=gen-small-thinking + +# ── Memory ── FIRST-BOOT values, expected to be tuned to a sane concurrency band +# (1.2-2.5x) by reading the boot log. A3B attention + fp8 KV is cheap, so 8 GiB likely +# over-provisions; pin down after the first boot reports its token count. +GEN_SMALL_GPU_MEM_UTIL=0.55 +GEN_SMALL_KV_CACHE_MEMORY=8589934592 +GEN_SMALL_MAX_MODEL_LEN=262144 +GEN_SMALL_MAX_NUM_SEQS=16 +GEN_SMALL_MAX_NUM_BATCHED_TOKENS=4096 +GEN_SMALL_KV_CACHE_DTYPE=fp8 + +# ── Speculative decoding ── MTP k=3. ⚠ Verify acceptance on THIS build (abliteration can +# desync MTP; MoE MTP has its own failure modes). Delete both --speculative-config lines +# in the compose to disable. +GEN_SMALL_SPEC_CONFIG={"method": "qwen3_5_mtp", "num_speculative_tokens": 3} + +GEN_SMALL_REASONING_PARSER=qwen3 +GEN_SMALL_REASONING_EFFORT=medium +GEN_SMALL_TOOL_PARSER=qwen3_xml +GEN_SMALL_ALLOC_CONF= diff --git a/stacks/gen-small-seat/compose.yaml b/stacks/gen-small-seat/compose.yaml new file mode 100644 index 0000000..c4c073b --- /dev/null +++ b/stacks/gen-small-seat/compose.yaml @@ -0,0 +1,115 @@ +# gen-small-seat — a fast, cheap "gen-small" seat: Qwen3.6-35B-A3B (3B active MoE), +# Heretic-abliterated, on fv-ml1 GPU 0, :8026. +# +# WHY: a tiny-active (A3B = ~3B active of 35B total) gen tier for high-volume / low-latency +# work, complementing the big gen seat (flash-next :8022). Operator-directed 2026-09-14. +# "3.8 if possible, 3.6 if not" — there is NO general Qwen3.8 A3B (the 3.8 MoEs are +# Flash-Next and the 2.4T), so this is the 3.6-35B-A3B fallback, which is the current-gen A3B. +# +# MODEL: /tank/aimodels/qwen36-35b-a3b-heretic-nvfp4 = +# llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-NVFP4-Experts-Only +# - Heretic v1.3.0 (MPOA — Magnitude-Preserving Orthogonal Ablation), the operator's +# house-favorite abliteration method: 88% fewer refusals (10/100 vs 83/100) at +# 0.0015 KL divergence (quality preserved). +# - modelopt NVFP4, experts-only (256 experts NVFP4; attn/shared-expert/linear_attn bf16). +# - 19 MTP tensors preserved (native), so speculative decoding is available. +# - hybrid arch (Qwen3_5MoeForConditionalGeneration): linear_attn (GDN) + attention + +# MoE, native context 262144. +# +# ⚠ NVFP4 MoE serving: this is modelopt NVFP4 on the experts. If the NVFP4 MoE kernel +# refuses ("does not support the deployment configuration ... kernel does not support +# current device"), the erp-seat lesson applies (Marlin fallback) — but this is Blackwell +# sm120 with native FP4, so it should take the native path. Read the boot log. +# +# ⚠ MTP on an A3B MoE is UNVERIFIED on this build — abliteration can desync an MTP head, and +# MoE MTP has its own failure modes. GEN_SMALL_SPEC_CONFIG defaults to qwen3_5_mtp k=3; +# gate it on a MEASURED acceptance (>=~40%) via a coherence + acceptance smoke, exactly as +# cyberprev/flash-next were. To turn MTP off, delete the two --speculative-config lines. +name: gen-small-seat + +services: + vllm-gen-small: + image: ${GEN_SMALL_IMAGE:-vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013} + container_name: ${GEN_SMALL_CONTAINER_NAME:-vllm-gen-small} + restart: unless-stopped + ipc: host + ports: + - "${GEN_SMALL_PORT:-8026}:8000" + volumes: + - /tank/aimodels/huggingface:/hfcache + - ${GEN_SMALL_MODEL:-/tank/aimodels/qwen36-35b-a3b-heretic-nvfp4}:/model:ro + environment: + - VLLM_API_KEY=${API_KEY:-} + - PYTORCH_CUDA_ALLOC_CONF=${GEN_SMALL_ALLOC_CONF:-} + command: + - /model + - --served-model-name + - ${GEN_SMALL_SERVED_NAME:-gen-small} + - ${GEN_SMALL_SERVED_NAME_THINK:-gen-small-thinking} + - --host + - 0.0.0.0 + - --port + - "8000" + - --quantization + - ${GEN_SMALL_QUANT:-modelopt_fp4} + - --gpu-memory-utilization + - "${GEN_SMALL_GPU_MEM_UTIL:-0.55}" + # KV pinned in bytes — GPU 0 is shared with cyberprev, so an explicit KV budget + # avoids the util-fraction fight between co-resident seats. fp8 KV + an A3B's small + # attention footprint make this cheap; tuned to a sane concurrency band, not max. + - --kv-cache-memory + - "${GEN_SMALL_KV_CACHE_MEMORY:-8589934592}" + - --max-model-len + - "${GEN_SMALL_MAX_MODEL_LEN:-262144}" + - --max-num-seqs + - "${GEN_SMALL_MAX_NUM_SEQS:-16}" + - --max-num-batched-tokens + - "${GEN_SMALL_MAX_NUM_BATCHED_TOKENS:-4096}" + - --trust-remote-code + - --dtype + - auto + # Hybrid backbone carries linear_attn (GDN); its state wants float32. + - --mamba-cache-dtype + - float32 + - --kv-cache-dtype + - ${GEN_SMALL_KV_CACHE_DTYPE:-fp8} + - --enable-prefix-caching + - --enable-chunked-prefill + - --reasoning-parser + - ${GEN_SMALL_REASONING_PARSER:-qwen3} + - --default-chat-template-kwargs + - '{"reasoning_effort": "${GEN_SMALL_REASONING_EFFORT:-medium}"}' + - --enable-auto-tool-choice + - --tool-call-parser + - ${GEN_SMALL_TOOL_PARSER:-qwen3_xml} + # MTP k=3 — verify acceptance on THIS build before trusting it (see header). + - --speculative-config + - '${GEN_SMALL_SPEC_CONFIG:-{"method": "qwen3_5_mtp", "num_speculative_tokens": 3}}' + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: + - "${GEN_SMALL_GPU_ID:-0}" + capabilities: + - gpu + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8000/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 900s + networks: + - tnet + labels: + - homepage.group=AI - Inference + - homepage.name=gen-small (Qwen3.6-35B-A3B heretic) + - homepage.icon=mdi-lightning-bolt + - homepage.description=Fast A3B (3B-active) Heretic-abliterated gen seat, MTP, fv-ml1 GPU 0 + - homepage.href=http://10.251.50.54:${GEN_SMALL_PORT:-8026}/docs + +networks: + tnet: + name: traefik-net + external: true diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index 0552011..11b4e77 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -70,8 +70,8 @@ model_list: # silently ignored in config per litellm #15020/#5524). Keep api_base in sync above. - model_name: summarizer litellm_params: - model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.251.50.54:8015/v1 + model: hosted_vllm/gen-small + api_base: http://10.251.50.54:8026/v1 api_key: os.environ/VLLM_API_KEY temperature: 0 extra_body: @@ -84,8 +84,8 @@ model_list: # + triage endpoint; same backend as summarizer. Keep api_base in sync above. - model_name: classifier litellm_params: - model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.251.50.54:8015/v1 + model: hosted_vllm/gen-small + api_base: http://10.251.50.54:8026/v1 api_key: os.environ/VLLM_API_KEY temperature: 0 extra_body: @@ -97,10 +97,23 @@ model_list: # alias: summarizer-large -> gen / qwen3.8-27b-uncensored (operator 2026-07-05). For heavier # summarization that wants the 35B-A3B heretic `gen` model instead of granite-8b. Thinking OFF # (matches gen). Keep api_base (:8015) + enable_thinking in sync with the gen record below. + # classifier-large -> gen-large (flash-next :8022). The accuracy tier for classification + # that a 3B-active seat may get wrong; classifier (above) is the fast gen-small default. + - model_name: classifier-large + litellm_params: + model: hosted_vllm/qwen3.8-flash-next-uncensored + api_base: http://10.251.50.54:8022/v1 + api_key: os.environ/VLLM_API_KEY + temperature: 0 + extra_body: + chat_template_kwargs: + enable_thinking: false + model_info: + mode: chat - model_name: summarizer-large litellm_params: - model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.251.50.54:8015/v1 + model: hosted_vllm/qwen3.8-flash-next-uncensored + api_base: http://10.251.50.54:8022/v1 api_key: os.environ/VLLM_API_KEY presence_penalty: 1.5 temperature: 0.7 @@ -122,8 +135,8 @@ model_list: # back to :8014 + model hosted_vllm/qwen-image-bench. --- - model_name: qwen-image-bench litellm_params: - model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.251.50.54:8015/v1 + model: hosted_vllm/qwen3.8-flash-next-uncensored + api_base: http://10.251.50.54:8022/v1 api_key: os.environ/VLLM_API_KEY temperature: 0 top_p: 1.0 @@ -136,8 +149,8 @@ model_list: mode: chat - model_name: image-judge litellm_params: - model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.251.50.54:8015/v1 + model: hosted_vllm/qwen3.8-flash-next-uncensored + api_base: http://10.251.50.54:8022/v1 api_key: os.environ/VLLM_API_KEY temperature: 0 top_p: 1.0 @@ -181,8 +194,8 @@ model_list: # than hand-tuned. - model_name: gen litellm_params: - model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.251.50.54:8015/v1 + model: hosted_vllm/qwen3.8-flash-next-uncensored + api_base: http://10.251.50.54:8022/v1 api_key: os.environ/VLLM_API_KEY presence_penalty: 1.5 temperature: 0.7 @@ -199,8 +212,8 @@ model_list: litellm_params: # Distinct served-name so a thinking-off `gen` request can't mutate this deployment's # enable_thinking (shared-config-mutation footgun). Same backend :8015, different model id. - model: hosted_vllm/qwen3.8-27b-uncensored-thinking - api_base: http://10.251.50.54:8015/v1 + model: hosted_vllm/qwen3.8-flash-next-uncensored-thinking + api_base: http://10.251.50.54:8022/v1 api_key: os.environ/VLLM_API_KEY # CANONICAL Qwen3.8 THINKING sampling set (Qwen + unsloth "Best Practices" # §1, identical in both): temperature=1.0, top_p=0.95, top_k=20, @@ -390,7 +403,7 @@ model_list: enable_thinking: false model_info: mode: chat - # sec-reasoning -> the SAME seat, thinking ON. Distinct served-name so a + # sec-reasoning -> the SAME seat (cyberprev, promoted 2026-09-14), thinking ON. Distinct served-name so a # thinking-off request can't mutate this deployment's enable_thinking (the # shared-config clobber). Canonical Qwen3.8 thinking samplers (temp 1.0/top_p 0.95). - model_name: sec-reasoning @@ -462,8 +475,8 @@ model_list: # temperature 0, so this matches the tested condition. - model_name: chat-judge litellm_params: - model: hosted_vllm/qwen3.8-27b-uncensored - api_base: http://10.251.50.54:8015/v1 + model: hosted_vllm/qwen3.8-flash-next-uncensored + api_base: http://10.251.50.54:8022/v1 api_key: os.environ/VLLM_API_KEY temperature: 0 top_p: 1.0 @@ -969,7 +982,34 @@ model_list: model_info: mode: chat - # cyberprev-27b -> hotdogs/Qwen3.8-27B-abliterated-cyber-preview (:8025, fv-ml1 GPU0). + # gen-small / gen-small-reasoning -> Qwen3.6-35B-A3B Heretic (llmfan46), fv-ml1 GPU0 :8026. + # Fast A3B (3B active) gen tier; MTP k=3 (measured 69.6% accept / 3.09 len). Backs the + # summarizer + classifier aliases too. Non-thinking + thinking split like the other seats. + - model_name: gen-small + litellm_params: + model: hosted_vllm/gen-small + api_base: http://10.251.50.54:8026/v1 + api_key: os.environ/VLLM_API_KEY + temperature: 0.7 + top_p: 0.8 + extra_body: + chat_template_kwargs: + enable_thinking: false + model_info: + mode: chat + - model_name: gen-small-reasoning + litellm_params: + model: hosted_vllm/gen-small-thinking + api_base: http://10.251.50.54:8026/v1 + api_key: os.environ/VLLM_API_KEY + temperature: 1.0 + top_p: 0.95 + extra_body: + chat_template_kwargs: + enable_thinking: true + model_info: + mode: chat + general_settings: master_key: os.environ/LITELLM_MASTER_KEY database_url: os.environ/DATABASE_URL