vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(fv-ml1): add gen-small A3B seat; sec/gen/char to native 262K; coder reclaim; gateway

Browse Source 
Full seat rebalance across GPU0/GPU1 (flash on GPU2 and empty GPU3 untouched),
operator-directed. Every target seat now serves native 262,144 context with
concurrency in the requested 1.2-2.5x band, verified from live boot logs:

  cyberprev (sec)   262144 @ 1.37x   depth-probed CLEAN to 259,722 tokens
  flash-next (gen)  262144 @ 1.31x   (untouched, already in band)
  gen-small (NEW)   262144 @ 2.56x   MTP k=3 measured 69.6% accept / 3.09 len
  char-rp           262144 @ 1.22x   (was 1.02x; util 0.40->0.52)
  char-rp-fast      262144 @ 2.04x   (util cap 0.30->0.24, pinned KV unchanged)

- gen-small: NEW seat, Qwen3.6-35B-A3B (3B active MoE), llmfan46 Heretic
  (MPOA) NVFP4 experts-only, already on disk at qwen36-35b-a3b-heretic-nvfp4.
  There is no general Qwen3.8 A3B (3.8 MoEs are Flash-Next and the 2.4T), so
  this is the 3.6 fallback the operator specified. GPU0, :8026, MTP k=3, coherent
  and MTP-verified before wiring. gen-small / gen-small-reasoning gateway aliases.

- coder: 8192 @ 13.75x -> 16384 @ 4.70x (util 0.06->0.055). Context doubled,
  waste cut. Not the exact 2-3x target: the 1.5B weight+overhead floor (~4.2 GiB)
  sits just under the util knob's resolution, so hitting <=3x reliably needs a
  --kv-cache-memory byte pin (compose change) rather than the util fraction.

- cyberprev raised 163840 -> 262144: depth-probed with non-repeating prompts to
  259,722 tokens, clean (no OOM, memory flat). Unlike mog-sec (same base arch,
  capped at 163840 for depth crashes), this checkpoint holds native depth.

- Gateway (operator calls): summarizer + classifier -> gen-small; new
  classifier-large -> gen-large (flash) for the accuracy tier; summarizer-large
  stays on flash. All verified end-to-end.

- GPU1 hit its ceiling raising char-rp; resolved by trimming char-rp-fast's
  reservation cap (its KV is pinned, so concurrency held at 2.04x) rather than
  moving a utility seat -- the shared GPU_ID on reward/embed/rerank made a
  single-seat move messier than the in-GPU rebalance.

Seat inventory regenerated from the live containers.
This commit is contained in:
vh
2026-09-14 10:25:10 -07:00
parent ff8dfed498
commit 05f3d3b2a3
4 changed files with 233 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/gen-small-seat/.env.example
+38
View File
@@ -0,0 +1,38 @@
# gen-small-seat — Qwen3.6-35B-A3B Heretic (llmfan46), fv-ml1 GPU 0, :8026.
# Copy to .env on the host at /opt/docker/compose/gen-small-seat/.env.
GEN_SMALL_IMAGE=vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013
API_KEY=
# ── Placement ── shares GPU 0 with cyberprev (sec). KV pinned in bytes so the two
# co-resident seats don't fight over a util fraction.
GEN_SMALL_GPU_ID=0
GEN_SMALL_PORT=8026
GEN_SMALL_CONTAINER_NAME=vllm-gen-small
# ── Model ── llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-NVFP4-Experts-Only
# Heretic v1.3.0 (MPOA); modelopt NVFP4 experts-only; 19 MTP preserved.
GEN_SMALL_MODEL=/tank/aimodels/qwen36-35b-a3b-heretic-nvfp4
GEN_SMALL_QUANT=modelopt_fp4
GEN_SMALL_SERVED_NAME=gen-small
GEN_SMALL_SERVED_NAME_THINK=gen-small-thinking
# ── Memory ── FIRST-BOOT values, expected to be tuned to a sane concurrency band
# (1.2-2.5x) by reading the boot log. A3B attention + fp8 KV is cheap, so 8 GiB likely
# over-provisions; pin down after the first boot reports its token count.
GEN_SMALL_GPU_MEM_UTIL=0.55
GEN_SMALL_KV_CACHE_MEMORY=8589934592
GEN_SMALL_MAX_MODEL_LEN=262144
GEN_SMALL_MAX_NUM_SEQS=16
GEN_SMALL_MAX_NUM_BATCHED_TOKENS=4096
GEN_SMALL_KV_CACHE_DTYPE=fp8
# ── Speculative decoding ── MTP k=3. ⚠ Verify acceptance on THIS build (abliteration can
# desync MTP; MoE MTP has its own failure modes). Delete both --speculative-config lines
# in the compose to disable.
GEN_SMALL_SPEC_CONFIG={"method": "qwen3_5_mtp", "num_speculative_tokens": 3}
GEN_SMALL_REASONING_PARSER=qwen3
GEN_SMALL_REASONING_EFFORT=medium
GEN_SMALL_TOOL_PARSER=qwen3_xml
GEN_SMALL_ALLOC_CONF=
stacks/gen-small-seat/compose.yaml
+115
View File
@@ -0,0 +1,115 @@
# gen-small-seat — a fast, cheap "gen-small" seat: Qwen3.6-35B-A3B (3B active MoE),
# Heretic-abliterated, on fv-ml1 GPU 0, :8026.
#
# WHY: a tiny-active (A3B = ~3B active of 35B total) gen tier for high-volume / low-latency
# work, complementing the big gen seat (flash-next :8022). Operator-directed 2026-09-14.
# "3.8 if possible, 3.6 if not" — there is NO general Qwen3.8 A3B (the 3.8 MoEs are
# Flash-Next and the 2.4T), so this is the 3.6-35B-A3B fallback, which is the current-gen A3B.
#
# MODEL: /tank/aimodels/qwen36-35b-a3b-heretic-nvfp4 =
# llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-NVFP4-Experts-Only
# - Heretic v1.3.0 (MPOA — Magnitude-Preserving Orthogonal Ablation), the operator's
# house-favorite abliteration method: 88% fewer refusals (10/100 vs 83/100) at
# 0.0015 KL divergence (quality preserved).
# - modelopt NVFP4, experts-only (256 experts NVFP4; attn/shared-expert/linear_attn bf16).
# - 19 MTP tensors preserved (native), so speculative decoding is available.
# - hybrid arch (Qwen3_5MoeForConditionalGeneration): linear_attn (GDN) + attention +
# MoE, native context 262144.
#
# ⚠ NVFP4 MoE serving: this is modelopt NVFP4 on the experts. If the NVFP4 MoE kernel
# refuses ("does not support the deployment configuration ... kernel does not support
# current device"), the erp-seat lesson applies (Marlin fallback) — but this is Blackwell
# sm120 with native FP4, so it should take the native path. Read the boot log.
#
# ⚠ MTP on an A3B MoE is UNVERIFIED on this build — abliteration can desync an MTP head, and
# MoE MTP has its own failure modes. GEN_SMALL_SPEC_CONFIG defaults to qwen3_5_mtp k=3;
# gate it on a MEASURED acceptance (>=~40%) via a coherence + acceptance smoke, exactly as
# cyberprev/flash-next were. To turn MTP off, delete the two --speculative-config lines.
name: gen-small-seat
services:
vllm-gen-small:
image: ${GEN_SMALL_IMAGE:-vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013}
container_name: ${GEN_SMALL_CONTAINER_NAME:-vllm-gen-small}
restart: unless-stopped
ipc: host
ports:
- "${GEN_SMALL_PORT:-8026}:8000"
volumes:
- /tank/aimodels/huggingface:/hfcache
- ${GEN_SMALL_MODEL:-/tank/aimodels/qwen36-35b-a3b-heretic-nvfp4}:/model:ro
environment:
- VLLM_API_KEY=${API_KEY:-}
- PYTORCH_CUDA_ALLOC_CONF=${GEN_SMALL_ALLOC_CONF:-}
command:
- /model
- --served-model-name
- ${GEN_SMALL_SERVED_NAME:-gen-small}
- ${GEN_SMALL_SERVED_NAME_THINK:-gen-small-thinking}
- --host
- 0.0.0.0
- --port
- "8000"
- --quantization
- ${GEN_SMALL_QUANT:-modelopt_fp4}
- --gpu-memory-utilization
- "${GEN_SMALL_GPU_MEM_UTIL:-0.55}"
# KV pinned in bytes — GPU 0 is shared with cyberprev, so an explicit KV budget
# avoids the util-fraction fight between co-resident seats. fp8 KV + an A3B's small
# attention footprint make this cheap; tuned to a sane concurrency band, not max.
- --kv-cache-memory
- "${GEN_SMALL_KV_CACHE_MEMORY:-8589934592}"
- --max-model-len
- "${GEN_SMALL_MAX_MODEL_LEN:-262144}"
- --max-num-seqs
- "${GEN_SMALL_MAX_NUM_SEQS:-16}"
- --max-num-batched-tokens
- "${GEN_SMALL_MAX_NUM_BATCHED_TOKENS:-4096}"
- --trust-remote-code
- --dtype
- auto
# Hybrid backbone carries linear_attn (GDN); its state wants float32.
- --mamba-cache-dtype
- float32
- --kv-cache-dtype
- ${GEN_SMALL_KV_CACHE_DTYPE:-fp8}
- --enable-prefix-caching
- --enable-chunked-prefill
- --reasoning-parser
- ${GEN_SMALL_REASONING_PARSER:-qwen3}
- --default-chat-template-kwargs
- '{"reasoning_effort": "${GEN_SMALL_REASONING_EFFORT:-medium}"}'
- --enable-auto-tool-choice
- --tool-call-parser
- ${GEN_SMALL_TOOL_PARSER:-qwen3_xml}
# MTP k=3 — verify acceptance on THIS build before trusting it (see header).
- --speculative-config
- '${GEN_SMALL_SPEC_CONFIG:-{"method": "qwen3_5_mtp", "num_speculative_tokens": 3}}'
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids:
- "${GEN_SMALL_GPU_ID:-0}"
capabilities:
- gpu
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 900s
networks:
- tnet
labels:
- homepage.group=AI - Inference
- homepage.name=gen-small (Qwen3.6-35B-A3B heretic)
- homepage.icon=mdi-lightning-bolt
- homepage.description=Fast A3B (3B-active) Heretic-abliterated gen seat, MTP, fv-ml1 GPU 0
- homepage.href=http://10.251.50.54:${GEN_SMALL_PORT:-8026}/docs
networks:
tnet:
name: traefik-net
external: true
stacks/litellm/conf/config.yaml
+58 -18
View File
@@ -70,8 +70,8 @@ model_list:
# silently ignored in config per litellm #15020/#5524). Keep api_base in sync above.
- model_name: summarizer
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.251.50.54:8015/v1
model: hosted_vllm/gen-small
api_base: http://10.251.50.54:8026/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0
extra_body:
@@ -84,8 +84,8 @@ model_list:
# + triage endpoint; same backend as summarizer. Keep api_base in sync above.
- model_name: classifier
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.251.50.54:8015/v1
model: hosted_vllm/gen-small
api_base: http://10.251.50.54:8026/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0
extra_body:
@@ -97,10 +97,23 @@ model_list:
# alias: summarizer-large -> gen / qwen3.8-27b-uncensored (operator 2026-07-05). For heavier
# summarization that wants the 35B-A3B heretic `gen` model instead of granite-8b. Thinking OFF
# (matches gen). Keep api_base (:8015) + enable_thinking in sync with the gen record below.
# classifier-large -> gen-large (flash-next :8022). The accuracy tier for classification
# that a 3B-active seat may get wrong; classifier (above) is the fast gen-small default.
- model_name: classifier-large
litellm_params:
model: hosted_vllm/qwen3.8-flash-next-uncensored
api_base: http://10.251.50.54:8022/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0
extra_body:
chat_template_kwargs:
enable_thinking: false
model_info:
mode: chat
- model_name: summarizer-large
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.251.50.54:8015/v1
model: hosted_vllm/qwen3.8-flash-next-uncensored
api_base: http://10.251.50.54:8022/v1
api_key: os.environ/VLLM_API_KEY
presence_penalty: 1.5
temperature: 0.7
@@ -122,8 +135,8 @@ model_list:
# back to :8014 + model hosted_vllm/qwen-image-bench. ---
- model_name: qwen-image-bench
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.251.50.54:8015/v1
model: hosted_vllm/qwen3.8-flash-next-uncensored
api_base: http://10.251.50.54:8022/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0
top_p: 1.0
@@ -136,8 +149,8 @@ model_list:
mode: chat
- model_name: image-judge
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.251.50.54:8015/v1
model: hosted_vllm/qwen3.8-flash-next-uncensored
api_base: http://10.251.50.54:8022/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0
top_p: 1.0
@@ -181,8 +194,8 @@ model_list:
# than hand-tuned.
- model_name: gen
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.251.50.54:8015/v1
model: hosted_vllm/qwen3.8-flash-next-uncensored
api_base: http://10.251.50.54:8022/v1
api_key: os.environ/VLLM_API_KEY
presence_penalty: 1.5
temperature: 0.7
@@ -199,8 +212,8 @@ model_list:
litellm_params:
# Distinct served-name so a thinking-off `gen` request can't mutate this deployment's
# enable_thinking (shared-config-mutation footgun). Same backend :8015, different model id.
model: hosted_vllm/qwen3.8-27b-uncensored-thinking
api_base: http://10.251.50.54:8015/v1
model: hosted_vllm/qwen3.8-flash-next-uncensored-thinking
api_base: http://10.251.50.54:8022/v1
api_key: os.environ/VLLM_API_KEY
# CANONICAL Qwen3.8 THINKING sampling set (Qwen + unsloth "Best Practices"
# §1, identical in both): temperature=1.0, top_p=0.95, top_k=20,
@@ -390,7 +403,7 @@ model_list:
enable_thinking: false
model_info:
mode: chat
# sec-reasoning -> the SAME seat, thinking ON. Distinct served-name so a
# sec-reasoning -> the SAME seat (cyberprev, promoted 2026-09-14), thinking ON. Distinct served-name so a
# thinking-off request can't mutate this deployment's enable_thinking (the
# shared-config clobber). Canonical Qwen3.8 thinking samplers (temp 1.0/top_p 0.95).
- model_name: sec-reasoning
@@ -462,8 +475,8 @@ model_list:
# temperature 0, so this matches the tested condition.
- model_name: chat-judge
litellm_params:
model: hosted_vllm/qwen3.8-27b-uncensored
api_base: http://10.251.50.54:8015/v1
model: hosted_vllm/qwen3.8-flash-next-uncensored
api_base: http://10.251.50.54:8022/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0
top_p: 1.0
@@ -969,7 +982,34 @@ model_list:
model_info:
mode: chat
# cyberprev-27b -> hotdogs/Qwen3.8-27B-abliterated-cyber-preview (:8025, fv-ml1 GPU0).
# gen-small / gen-small-reasoning -> Qwen3.6-35B-A3B Heretic (llmfan46), fv-ml1 GPU0 :8026.
# Fast A3B (3B active) gen tier; MTP k=3 (measured 69.6% accept / 3.09 len). Backs the
# summarizer + classifier aliases too. Non-thinking + thinking split like the other seats.
- model_name: gen-small
litellm_params:
model: hosted_vllm/gen-small
api_base: http://10.251.50.54:8026/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0.7
top_p: 0.8
extra_body:
chat_template_kwargs:
enable_thinking: false
model_info:
mode: chat
- model_name: gen-small-reasoning
litellm_params:
model: hosted_vllm/gen-small-thinking
api_base: http://10.251.50.54:8026/v1
api_key: os.environ/VLLM_API_KEY
temperature: 1.0
top_p: 0.95
extra_body:
chat_template_kwargs:
enable_thinking: true
model_info:
mode: chat
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
database_url: os.environ/DATABASE_URL