feat(fv-ml1): add gen-small A3B seat; sec/gen/char to native 262K; coder reclaim; gateway

Full seat rebalance across GPU0/GPU1 (flash on GPU2 and empty GPU3 untouched), operator-directed. Every target seat now serves native 262,144 context with concurrency in the requested 1.2-2.5x band, verified from live boot logs: cyberprev (sec) 262144 @ 1.37x depth-probed CLEAN to 259,722 tokens flash-next (gen) 262144 @ 1.31x (untouched, already in band) gen-small (NEW) 262144 @ 2.56x MTP k=3 measured 69.6% accept / 3.09 len char-rp 262144 @ 1.22x (was 1.02x; util 0.40->0.52) char-rp-fast 262144 @ 2.04x (util cap 0.30->0.24, pinned KV unchanged) - gen-small: NEW seat, Qwen3.6-35B-A3B (3B active MoE), llmfan46 Heretic (MPOA) NVFP4 experts-only, already on disk at qwen36-35b-a3b-heretic-nvfp4. There is no general Qwen3.8 A3B (3.8 MoEs are Flash-Next and the 2.4T), so this is the 3.6 fallback the operator specified. GPU0, :8026, MTP k=3, coherent and MTP-verified before wiring. gen-small / gen-small-reasoning gateway aliases. - coder: 8192 @ 13.75x -> 16384 @ 4.70x (util 0.06->0.055). Context doubled, waste cut. Not the exact 2-3x target: the 1.5B weight+overhead floor (~4.2 GiB) sits just under the util knob's resolution, so hitting <=3x reliably needs a --kv-cache-memory byte pin (compose change) rather than the util fraction. - cyberprev raised 163840 -> 262144: depth-probed with non-repeating prompts to 259,722 tokens, clean (no OOM, memory flat). Unlike mog-sec (same base arch, capped at 163840 for depth crashes), this checkpoint holds native depth. - Gateway (operator calls): summarizer + classifier -> gen-small; new classifier-large -> gen-large (flash) for the accuracy tier; summarizer-large stays on flash. All verified end-to-end. - GPU1 hit its ceiling raising char-rp; resolved by trimming char-rp-fast's reservation cap (its KV is pinned, so concurrency held at 2.04x) rather than moving a utility seat -- the shared GPU_ID on reward/embed/rerank made a single-seat move messier than the in-GPU rebalance. Seat inventory regenerated from the live containers.
2026-09-14 10:25:10 -07:00
parent ff8dfed498
commit 05f3d3b2a3
4 changed files with 233 additions and 27 deletions
@@ -70,8 +70,8 @@ model_list:
  #   silently ignored in config per litellm #15020/#5524). Keep api_base in sync above.
  - model_name: summarizer
    litellm_params:
-      model: hosted_vllm/qwen3.8-27b-uncensored
-      api_base: http://10.251.50.54:8015/v1
+      model: hosted_vllm/gen-small
+      api_base: http://10.251.50.54:8026/v1
      api_key: os.environ/VLLM_API_KEY
      temperature: 0
      extra_body:
@@ -84,8 +84,8 @@ model_list:
  #   + triage endpoint; same backend as summarizer. Keep api_base in sync above.
  - model_name: classifier
    litellm_params:
-      model: hosted_vllm/qwen3.8-27b-uncensored
-      api_base: http://10.251.50.54:8015/v1
+      model: hosted_vllm/gen-small
+      api_base: http://10.251.50.54:8026/v1
      api_key: os.environ/VLLM_API_KEY
      temperature: 0
      extra_body:
@@ -97,10 +97,23 @@ model_list:
  # alias: summarizer-large -> gen / qwen3.8-27b-uncensored (operator 2026-07-05). For heavier
  #   summarization that wants the 35B-A3B heretic `gen` model instead of granite-8b. Thinking OFF
  #   (matches gen). Keep api_base (:8015) + enable_thinking in sync with the gen record below.
+  # classifier-large -> gen-large (flash-next :8022). The accuracy tier for classification
+  #   that a 3B-active seat may get wrong; classifier (above) is the fast gen-small default.
+  - model_name: classifier-large
+    litellm_params:
+      model: hosted_vllm/qwen3.8-flash-next-uncensored
+      api_base: http://10.251.50.54:8022/v1
+      api_key: os.environ/VLLM_API_KEY
+      temperature: 0
+      extra_body:
+        chat_template_kwargs:
+          enable_thinking: false
+    model_info:
+      mode: chat
  - model_name: summarizer-large
    litellm_params:
-      model: hosted_vllm/qwen3.8-27b-uncensored
-      api_base: http://10.251.50.54:8015/v1
+      model: hosted_vllm/qwen3.8-flash-next-uncensored
+      api_base: http://10.251.50.54:8022/v1
      api_key: os.environ/VLLM_API_KEY
      presence_penalty: 1.5
      temperature: 0.7
@@ -122,8 +135,8 @@ model_list:
  #     back to :8014 + model hosted_vllm/qwen-image-bench. ---
  - model_name: qwen-image-bench
    litellm_params:
-      model: hosted_vllm/qwen3.8-27b-uncensored
-      api_base: http://10.251.50.54:8015/v1
+      model: hosted_vllm/qwen3.8-flash-next-uncensored
+      api_base: http://10.251.50.54:8022/v1
      api_key: os.environ/VLLM_API_KEY
      temperature: 0
      top_p: 1.0
@@ -136,8 +149,8 @@ model_list:
      mode: chat
  - model_name: image-judge
    litellm_params:
-      model: hosted_vllm/qwen3.8-27b-uncensored
-      api_base: http://10.251.50.54:8015/v1
+      model: hosted_vllm/qwen3.8-flash-next-uncensored
+      api_base: http://10.251.50.54:8022/v1
      api_key: os.environ/VLLM_API_KEY
      temperature: 0
      top_p: 1.0
@@ -181,8 +194,8 @@ model_list:
  # than hand-tuned.
  - model_name: gen
    litellm_params:
-      model: hosted_vllm/qwen3.8-27b-uncensored
-      api_base: http://10.251.50.54:8015/v1
+      model: hosted_vllm/qwen3.8-flash-next-uncensored
+      api_base: http://10.251.50.54:8022/v1
      api_key: os.environ/VLLM_API_KEY
      presence_penalty: 1.5
      temperature: 0.7
@@ -199,8 +212,8 @@ model_list:
    litellm_params:
      # Distinct served-name so a thinking-off `gen` request can't mutate this deployment's
      # enable_thinking (shared-config-mutation footgun). Same backend :8015, different model id.
-      model: hosted_vllm/qwen3.8-27b-uncensored-thinking
-      api_base: http://10.251.50.54:8015/v1
+      model: hosted_vllm/qwen3.8-flash-next-uncensored-thinking
+      api_base: http://10.251.50.54:8022/v1
      api_key: os.environ/VLLM_API_KEY
      # CANONICAL Qwen3.8 THINKING sampling set (Qwen + unsloth "Best Practices"
      # §1, identical in both): temperature=1.0, top_p=0.95, top_k=20,
@@ -390,7 +403,7 @@ model_list:
          enable_thinking: false
    model_info:
      mode: chat
-  # sec-reasoning -> the SAME seat, thinking ON. Distinct served-name so a
+  # sec-reasoning -> the SAME seat (cyberprev, promoted 2026-09-14), thinking ON. Distinct served-name so a
  #   thinking-off request can't mutate this deployment's enable_thinking (the
  #   shared-config clobber). Canonical Qwen3.8 thinking samplers (temp 1.0/top_p 0.95).
  - model_name: sec-reasoning
@@ -462,8 +475,8 @@ model_list:
  #   temperature 0, so this matches the tested condition.
  - model_name: chat-judge
    litellm_params:
-      model: hosted_vllm/qwen3.8-27b-uncensored
-      api_base: http://10.251.50.54:8015/v1
+      model: hosted_vllm/qwen3.8-flash-next-uncensored
+      api_base: http://10.251.50.54:8022/v1
      api_key: os.environ/VLLM_API_KEY
      temperature: 0
      top_p: 1.0
@@ -969,7 +982,34 @@ model_list:
    model_info:
      mode: chat

-  # cyberprev-27b -> hotdogs/Qwen3.8-27B-abliterated-cyber-preview (:8025, fv-ml1 GPU0).
+  # gen-small / gen-small-reasoning -> Qwen3.6-35B-A3B Heretic (llmfan46), fv-ml1 GPU0 :8026.
+  #   Fast A3B (3B active) gen tier; MTP k=3 (measured 69.6% accept / 3.09 len). Backs the
+  #   summarizer + classifier aliases too. Non-thinking + thinking split like the other seats.
+  - model_name: gen-small
+    litellm_params:
+      model: hosted_vllm/gen-small
+      api_base: http://10.251.50.54:8026/v1
+      api_key: os.environ/VLLM_API_KEY
+      temperature: 0.7
+      top_p: 0.8
+      extra_body:
+        chat_template_kwargs:
+          enable_thinking: false
+    model_info:
+      mode: chat
+  - model_name: gen-small-reasoning
+    litellm_params:
+      model: hosted_vllm/gen-small-thinking
+      api_base: http://10.251.50.54:8026/v1
+      api_key: os.environ/VLLM_API_KEY
+      temperature: 1.0
+      top_p: 0.95
+      extra_body:
+        chat_template_kwargs:
+          enable_thinking: true
+    model_info:
+      mode: chat
+
 general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL