diff --git a/services/flash-next-mtp-bench/README.md b/services/flash-next-mtp-bench/README.md new file mode 100644 index 0000000..a5b1eed --- /dev/null +++ b/services/flash-next-mtp-bench/README.md @@ -0,0 +1,106 @@ +# Does MTP actually hurt Qwen3.8-Flash-Next on one Blackwell card? + +Status: **campaign built, running.** Results and a verdict land here with the raw +JSON alongside, so the conclusion can be re-derived rather than taken on faith. + +## Why this exists + +The `flash-next-seat` stack shipped with MTP speculative decoding **off**, citing +vLLM's published recipe: on 4×H100 it measured MTP as worse at every concurrency +tested — 8–36% lower request throughput, 32–173% higher per-token latency, ~36% +acceptance — and says don't default it on. + +**That was not valid evidence about our seat, and defaulting on it was the wrong +call.** Our own measurement rule says the harness is part of the number and that +cross-harness comparisons are invalid, not merely noisy. The recipe's harness +differs from ours on nearly every axis that could plausibly drive the result: + +| | vLLM recipe | this seat | +|---|---|---| +| GPUs | 4× H100 80 GB (Hopper) | 1× RTX PRO 6000 96 GB (Blackwell, sm_120) | +| Parallelism | TP=4 — experts sharded 4 ways, all-reduce per layer | **TP=1 — every expert local, no collective** | +| PLE table | offloaded (forced: 80 GB cards can't hold it) | offloaded (chosen) | +| k tested | **3 only** | 1, 2, 3 | + +## Two mechanisms that could make MTP lose here, and one that makes k the real question + +Spec decoding's usual win is that verifying k+1 tokens costs about the same as +decoding 1, because decode is memory-bandwidth-bound: you read the weights once +either way. **Two things about this model break that assumption.** + +1. **Sparse-MoE expert-read amplification.** 512 experts, 10 active per token. At + batch 1, decoding one token touches ~10 experts. Verifying k+1 tokens routes + each position to *its own* 10, largely disjoint — so the weights read scale + with the token count instead of staying flat. The "free verification" premise + does not hold for an ultra-sparse MoE. This is worst at low concurrency and + shrinks as the batch already touches many experts anyway, which is exactly why + the sweep has to cross concurrency and not just report one number. +2. **PLE/UVA fetch amplification.** The 51B n-gram table lives in host RAM and is + read over PCIe. Every token position needs its own row lookups, so k+1 tokens + means k+1× the host round-trips — multiplying traffic on the single slowest + link in the system, the one we deliberately moved off the card. + +**And the mechanism that makes k the actual experiment:** the checkpoint's +`mtp_num_hidden_layers` is **1**. The draft head is a *single module run +autoregressively* for k>1 — the pattern our own quant playbook §5.1 flags, where +deeper k improves acceptance and destroys throughput. If throughput falls +monotonically in k while acceptance rises, then **k=1 may win and the recipe's +k=3 number says nothing about it.** Nobody has published k=1 for this model. + +So there are coherent reasons MTP could genuinely lose here — and an equally +coherent reason the published number is the wrong number to decide on. + +## Design + +Five boots. Arms in order: `off_A`, `k1`, `k2`, `k3`, `off_B`. + +- **Repeats** — 3 per (arm, concurrency) cell, concurrency ∈ {1, 4, 8}. Median + reported with spread, never a single run. +- **Noise floor** — `off_A` and `off_B` are the *same configuration*, booted first + and last. Their difference is the floor, and because they are separate boots it + includes boot-to-boot variance that three reps inside one boot cannot see. + Running `off_B` last also catches monotonic drift across the campaign. +- **Positive control (a)** — MTP acceptance must be **> 0** on every MTP arm, read + from `/metrics` as a delta. A head that loads uninitialised serves fine and + reports ~0% accept; that is the failure our `gen` seat's `re:^mtp.*` + ignore-list footgun produces. **An arm with ~0% acceptance is void** — it + measured a broken head, not MTP. (Pre-checked: this checkpoint excludes + `mtp.*` and `model.mtp.*` from quantization in both quant configs, and its own + audit reports all 31 MTP tensors byte-identical to source.) +- **Positive control (b)** — aggregate throughput must **rise with concurrency** + inside every arm. Known-true for a batching server; if the harness can't see + it, the harness is blind and its negatives are worthless. +- **Null control** — `off_A` vs `off_B` must show no effect beyond the floor. +- **Stated sensitivity floor** — the report prints "cannot resolve effects smaller + than X" from the observed `off_A`/`off_B` spread. A delta under it is not a + finding. + +**Harness parity.** Every arm's argv is *derived from the live compose file* via +`docker compose config`, not retyped, so the arms are provably identical except +for `--speculative-config`. That is what stops a stray flag from quietly becoming +the real independent variable. + +## Instruments + +- `concbench.py` — the **house** concurrency harness, reused unchanged from + `services/gen-seat-mixed-quant/bench/`. It already avoids the three traps that + have produced confident wrong answers here before: unseeded nonces so prefix + caching can't fake prefill, MTP acceptance read as a **delta** so a long-lived + seat's history doesn't swamp it, and aggregate throughput from wall clock rather + than a sum of per-request rates. +- `run-campaign.sh` — the driver. Runs on fv-ml1 against `127.0.0.1` so the + NH3↔FV mesh hop is not in the measurement. + +## Known gaps in this design + +- **No TTFT.** `concbench.py` doesn't stream, so prefill latency isn't measured + separately. The MTP mechanisms above are decode-side, so this is an acceptable + omission — but it means the campaign cannot speak to first-token latency. +- **One prompt shape.** ~400 output tokens from a short prompt. MTP acceptance is + workload-dependent; a long-context or code workload could differ and this says + nothing about them. +- **The offload's contribution is not isolated.** If MTP loses, this design cannot + separate mechanism 1 from mechanism 2. The clean discriminator is available and + cheap — re-run the best MTP arm at TP=2 across GPU 2+3 with `cpu_offload: false` + so the table is resident, and compare the MTP delta with and without the PCIe + path. GPU 3 is idle, so this is a follow-up worth doing if the answer matters. diff --git a/services/flash-next-mtp-bench/run-campaign.sh b/services/flash-next-mtp-bench/run-campaign.sh new file mode 100755 index 0000000..4cf3dc3 --- /dev/null +++ b/services/flash-next-mtp-bench/run-campaign.sh @@ -0,0 +1,165 @@ +#!/usr/bin/env bash +# run-campaign.sh — does MTP speculative decoding help or hurt Qwen3.8-Flash-Next +# on ONE RTX PRO 6000 with the n-gram table offloaded to host RAM? +# +# WHY THIS EXISTS. vLLM's published recipe measured MTP on 4xH100 as worse at +# every concurrency (8-36% less throughput, 32-173% more per-token latency, ~36% +# acceptance) and we initially defaulted MTP off on that basis. That was a +# cross-harness comparison and therefore not valid evidence about THIS seat: +# 4xH100 is TP=4 Hopper with experts sharded four ways and an all-reduce per +# layer; this is TP=1 Blackwell with every expert local. Our own rule says the +# harness is part of the number and cross-harness comparisons are invalid, not +# merely noisy. So we measure it here. +# +# AND THE RECIPE ONLY TESTED k=3. The checkpoint's mtp_num_hidden_layers is 1, so +# the draft head is a SINGLE module run autoregressively for k>1 (quant playbook +# §5.1: deeper k improves acceptance and destroys throughput). If throughput +# falls monotonically in k while acceptance rises, k=1 may well WIN and the +# recipe's k=3 number says nothing about it. That is the hypothesis this sweeps. +# +# ── MEASUREMENT DISCIPLINE ────────────────────────────────────────────────── +# Every number here carries repeats, a noise floor, a positive control and a null +# control, or it does not get to carry a conclusion. +# +# REPEATS 3 reps per (arm, concurrency) cell; median reported with spread. +# NOISE FLOOR The `off` arm is booted TWICE -- off_A first and off_B last. Both +# arms are the same configuration, so their difference IS the +# floor, and it includes boot-to-boot variance, which three reps +# inside one boot cannot see. Running off_B last also catches any +# monotonic drift across the campaign (thermals, page cache). +# POSITIVE CTL (a) MTP acceptance must be > 0 on every MTP arm. A head that +# loads uninitialised reports ~0% accept and still serves -- +# the exact failure our gen seat's `re:^mtp.*` ignore-list +# footgun produces. If acceptance is ~0 the throughput number +# is measuring a broken head, not MTP, and the arm is void. +# (b) Aggregate throughput must RISE with concurrency inside every +# arm. That is known-true for a batching server; if the +# harness cannot see it, the harness is blind and its nulls +# are worthless. +# NULL CONTROL off_A vs off_B must show no effect beyond the floor. +# FLOOR STATED The report prints "cannot resolve effects smaller than X" from +# the observed off_A/off_B spread. A delta under it is not a +# finding. +# +# HARNESS PARITY. Every arm's argv is DERIVED from the live compose file, so the +# arms are provably identical except for --speculative-config. Nothing is +# retyped, which is what stops a stray flag from becoming the real independent +# variable. +# +# The production container is stopped for the duration: same GPU, same port. The +# seat has no consumers yet, so this costs nothing. It is restored at the end. +set -uo pipefail + +STACK_DIR=${STACK_DIR:-/opt/docker/compose/flash-next-seat} +OUT=${OUT:-/tank/aimodels/flash-next-mtp-bench} +BENCH=${BENCH:-$OUT/concbench.py} +PORT=${PORT:-8022} +GPU=${GPU:-2} +REPS=${REPS:-3} +CONCS=${CONCS:-"1 4 8"} +MAXTOK=${MAXTOK:-400} +RPS=${RPS:-4} # requests per stream, per concbench +METHOD=${METHOD:-qwen4_exp_mtp} +MODEL=${MODEL:-qwen3.8-flash-next-uncensored} +NAME=fn-mtp-bench +BOOT_TIMEOUT=${BOOT_TIMEOUT:-2400} + +mkdir -p "$OUT" +log(){ echo "[$(date -u +%H:%M:%S)] $*" | tee -a "$OUT/campaign.log"; } + +# --- derive the production argv + run opts straight from the compose file ----- +read_compose() { + sudo -n docker compose -f "$STACK_DIR/compose.yaml" --env-file "$STACK_DIR/.env" config --format json +} +log "deriving argv from $STACK_DIR/compose.yaml" +read_compose > "$OUT/resolved-compose.json" || { log "FATAL: could not resolve compose"; exit 1; } +python3 - "$OUT/resolved-compose.json" "$OUT/argv.txt" "$OUT/envs.txt" <<'PY' +import json,sys +d=json.load(open(sys.argv[1])) +svc=d["services"]["vllm-flash-next"] +open(sys.argv[2],"w").write("\n".join(str(x) for x in svc["command"])+"\n") +env=svc.get("environment") or {} +if isinstance(env,dict): items=[(k,v) for k,v in env.items()] +else: items=[e.split("=",1) for e in env] +open(sys.argv[3],"w").write("\n".join(f"{k}={'' if v is None else v}" for k,v in items)+"\n") +PY +mapfile -t ARGV < "$OUT/argv.txt" +ENVARGS=(); while IFS= read -r l; do [ -n "$l" ] && ENVARGS+=(-e "$l"); done < "$OUT/envs.txt" +log "argv has ${#ARGV[@]} entries; ${#ENVARGS[@]} env flags" +printf '%s\n' "${ARGV[@]}" | sed 's/^/ /' >> "$OUT/campaign.log" + +MODEL_DIR=$(python3 -c " +import json,sys +d=json.load(open('$OUT/resolved-compose.json')) +for v in d['services']['vllm-flash-next']['volumes']: + t=v['target'] if isinstance(v,dict) else v.split(':')[1] + s=v['source'] if isinstance(v,dict) else v.split(':')[0] + if t=='/model': print(s) +") +HFCACHE=$(python3 -c " +import json +d=json.load(open('$OUT/resolved-compose.json')) +for v in d['services']['vllm-flash-next']['volumes']: + t=v['target'] if isinstance(v,dict) else v.split(':')[1] + s=v['source'] if isinstance(v,dict) else v.split(':')[0] + if t=='/hfcache': print(s) +") +IMAGE=$(python3 -c " +import json; print(json.load(open('$OUT/resolved-compose.json'))['services']['vllm-flash-next']['image'])") +log "image=$IMAGE model=$MODEL_DIR" + +stop_bench(){ sudo -n docker rm -f "$NAME" >/dev/null 2>&1 || true; } +trap 'stop_bench' EXIT + +boot(){ # boot [spec-json] + local arm="$1"; shift + local extra=(); [ $# -gt 0 ] && [ -n "${1:-}" ] && extra=(--speculative-config "$1") + stop_bench + log "=== boot arm=$arm ${extra[*]:-(no spec-config)}" + sudo -n docker run -d --name "$NAME" --ipc host --ulimit memlock=-1 \ + --gpus "\"device=$GPU\"" -p "$PORT:8000" \ + -v "$HFCACHE:/hfcache" -v "$MODEL_DIR:/model:ro" \ + "${ENVARGS[@]}" "$IMAGE" "${ARGV[@]}" "${extra[@]}" \ + > "$OUT/$arm.cid" 2>"$OUT/$arm.runerr" || { log " docker run FAILED: $(cat "$OUT/$arm.runerr")"; return 1; } + local t0=$SECONDS + while [ $((SECONDS-t0)) -lt "$BOOT_TIMEOUT" ]; do + if curl -sf "http://127.0.0.1:$PORT/health" >/dev/null 2>&1; then + log " healthy after $((SECONDS-t0))s" + sudo -n docker logs "$NAME" > "$OUT/$arm.boot.log" 2>&1 + return 0 + fi + if ! sudo -n docker ps --format '{{.Names}}' | grep -qx "$NAME"; then + log " CONTAINER DIED during boot"; sudo -n docker logs "$NAME" > "$OUT/$arm.boot.log" 2>&1 + tail -25 "$OUT/$arm.boot.log" | sed 's/^/ /' | tee -a "$OUT/campaign.log"; return 1 + fi + sleep 15 + done + log " BOOT TIMEOUT after ${BOOT_TIMEOUT}s"; sudo -n docker logs "$NAME" > "$OUT/$arm.boot.log" 2>&1; return 1 +} + +bench_arm(){ # bench_arm + local arm="$1" cargs=() + for c in $CONCS; do cargs+=(--concurrency "$c"); done + for rep in $(seq 1 "$REPS"); do + log " bench $arm rep=$rep" + python3 "$BENCH" --base "http://127.0.0.1:$PORT" --model "$MODEL" \ + "${cargs[@]}" --requests-per-stream "$RPS" --max-tokens "$MAXTOK" \ + --tag "$arm-rep$rep" --out "$OUT/res-$arm-rep$rep.json" 2>&1 | tee -a "$OUT/campaign.log" + done +} + +# --- the campaign ----------------------------------------------------------- +# off_A first, the MTP arms in ascending k, off_B LAST so the floor brackets the +# whole run rather than sitting at one end of it. +run_arm(){ boot "$1" "${2:-}" && bench_arm "$1" || log " arm $1 SKIPPED (boot failed)"; } + +log "########## CAMPAIGN START (reps=$REPS concs='$CONCS' max_tokens=$MAXTOK) ##########" +run_arm off_A "" +run_arm k1 "{\"method\":\"$METHOD\",\"num_speculative_tokens\":1}" +run_arm k2 "{\"method\":\"$METHOD\",\"num_speculative_tokens\":2}" +run_arm k3 "{\"method\":\"$METHOD\",\"num_speculative_tokens\":3}" +run_arm off_B "" +stop_bench +log "########## CAMPAIGN DONE -- results in $OUT ##########" +log "restoring the production container" +sudo -n docker compose -f "$STACK_DIR/compose.yaml" --env-file "$STACK_DIR/.env" up -d 2>&1 | tail -3 | tee -a "$OUT/campaign.log" diff --git a/stacks/flash-next-seat/.env.example b/stacks/flash-next-seat/.env.example index e850df4..3892dfb 100644 --- a/stacks/flash-next-seat/.env.example +++ b/stacks/flash-next-seat/.env.example @@ -37,21 +37,33 @@ FN_SERVED_NAME_THINK=qwen3.8-flash-next-uncensored-thinking # where host and device share one unified pool and "offload" frees nothing. FN_ENGRAM_CONFIG={"cpu_offload": true} -# ── Memory and context ────────────────────────────────────────────────────── -# FIRST-BOOT. Card is dedicated, so 0.90 (the vLLM recipe's own figure for offload -# configs) is not competing with a neighbour. Expected resident weights ~78 GiB of -# 95.6, leaving ~8-9 GiB for KV + activation inside the 0.90 budget. -# ⚠ NEXT STEP after first boot: read vLLM's budget line and replace this with an -# explicit --kv-cache-memory in BYTES. A ratio sizes the KV calculation but does -# not bound peak activation — that is what bit stacks/mog-sec twice on 2026-09-10. -FN_GPU_MEM_UTIL=0.90 +# Card is dedicated -- one tenant, nothing to compete with. Measured at 262K: +# weights 74.36 GiB resident, 560,654 KV tokens, 2.14x concurrency. +# ⚠ This ratio is now ADVISORY ONLY -- see the KV pin below, which overrides it. +FN_GPU_MEM_UTIL=0.96 +# ⚠⚠ 14.00 GiB, PINNED IN BYTES AND MEASURED THE HARD WAY (2026-09-13). +# 16.00 GiB was tried first and nearly OOM'd: a 155,497-token prefill drove GPU 2 to +# 97,074 of 97,887 MiB and the allocator logged "OOM on device 0 while trying to +# allocate 488636416 bytes (free: 422117376)" -- 466 MiB wanted, 403 MiB free. The +# request survived but that is one step before the mog-sec crash shape. +# ⚠ WHY THE ESTIMATE WAS WRONG: --kv-cache-memory makes vLLM SKIP MEMORY PROFILING +# and ignore --gpu-memory-utilization entirely. The profiler was the thing accounting +# for deep-prefill activation; pinning bytes turns it off. Do NOT take vLLM's +# "17.46 GiB to fully utilize" suggestion -- it is computed from a profile taken at +# max-num-batched-tokens depth and is 3.5 GiB above what a 150K request survives. +# Re-raising requires re-running the deep probe and reading the allocator log. +# VERIFIED at 14.00 GiB: 0 OOM warnings, 155,557-token prefill in 14.2 s, 2,085 MiB +# still free on the card at peak. +FN_KV_CACHE_MEMORY=15032385536 -# FIRST-BOOT. 128K, against a 262,144 native ceiling. Raising this is gated on a -# depth bisect with a NON-REPEATING prompt, because two open upstream issues make -# depth the risky axis: #54764 (PLE short-conv prefill pads every request in a -# batch to the batch-max query length) and #54919 (long prefill starving decode -# for 3-7 minutes). See the compose comment. -FN_MAX_MODEL_LEN=131072 +# FULL NATIVE 262,144 (operator-directed 2026-09-13). The KV pool holds ~641K +# tokens, so a single max-length request fits with ~2.4x concurrency to spare. +# ⚠ STARTUP IS NOT A DEPTH TEST. Two open upstream issues make depth the risky +# axis -- #54764 (PLE short-conv prefill pads every request in a batch to the +# batch-MAX query length) and #54919 (long prefill starving active decode for 3-7 +# minutes) -- and vLLM's own recipe admits a single 262K request was never tested. +# If deep requests misbehave, --max-num-batched-tokens is the lever, not this. +FN_MAX_MODEL_LEN=262144 FN_MAX_NUM_SEQS=16 FN_MAX_NUM_BATCHED_TOKENS=8192 FN_MAMBA_CACHE_MODE=align diff --git a/stacks/flash-next-seat/compose.yaml b/stacks/flash-next-seat/compose.yaml index babfe46..ed6117e 100644 --- a/stacks/flash-next-seat/compose.yaml +++ b/stacks/flash-next-seat/compose.yaml @@ -104,7 +104,56 @@ services: # KV calculation but does not bound peak activation, and it silently yields a # different cache depending on what is resident at start time. - --gpu-memory-utilization - - ${FN_GPU_MEM_UTIL:-0.90} + - ${FN_GPU_MEM_UTIL:-0.96} + # ⚠ KV PINNED IN BYTES, and the figure is MEASURED at 262K, not derived. + # The ratio above sizes the KV calculation but does NOT bound actual usage: + # peak activation is profiled once at startup and real long-context work + # exceeds the profile. stacks/mog-sec proved that twice on 2026-09-10, where + # vLLM's own budget line showed the seat running 0.9 GiB OVER its stated + # reservation. An explicit figure is reproducible; a ratio silently yields a + # different cache depending on what else is resident at start time. + # + # ⚠⚠ 16.00 GiB WAS TRIED FIRST AND WAS TOO AGGRESSIVE -- MEASURED, 2026-09-13. + # A 155,497-token non-repeating prefill drove GPU 2 to 97,074 of 97,887 MiB and + # the allocator logged + # OOM on device 0 while trying to allocate 488636416 bytes (free: 422117376) + # i.e. it wanted 466 MiB with 403 MiB left. The request still completed (the + # caching allocator recovered) but that is ONE STEP before the mog-sec crash + # shape: ~1.04 GiB wanted with ~600 MB free. 14.00 GiB restores ~2 GiB of + # headroom and was chosen over the 16.00 that nearly failed. + # + # ⚠ THE REASON THE FIRST ESTIMATE WAS WRONG, because it is not obvious: setting + # --kv-cache-memory makes vLLM SKIP MEMORY PROFILING ENTIRELY and ignore + # --gpu-memory-utilization ("reserved 16.0 GiB ... and skipped memory profiling. + # This does not respect the gpu_memory_utilization config"). So the 0.96 cap + # stops protecting anything, and the 1.65 GiB "peak activation" figure -- which + # was profiled against max-num-batched-tokens=8192, not against real depth -- + # stops being subtracted at all. The profiler was the thing keeping deep + # prefills safe, and pinning bytes turns it off. + # + # 15,032,385,536 B = 14.00 GiB. For reference the 262K boot's own accounting: + # free on device 94.42 GiB + # weights+non-torch 75.07 + # peak activation 1.65 (profiled against max-num-batched-tokens, + # 8192 here -- NOT against max-model-len, + # which is why doubling context to 262K left + # this number unchanged) + # CUDA graphs 0.10 + # → theoretical KV 17.60 GiB (on paper; NOT achievable at depth) + # + # ⚠ vLLM's own "--kv-cache-memory=18745235968 (17.46 GiB) to fully utilize gpu + # memory" suggestion is 3.5 GiB ABOVE what survives a deep prefill here. Do not + # take it. That line is computed from the startup profile, which is measured at + # max-num-batched-tokens depth and says nothing about a 150K-token request. + # Open upstream #54764 compounds it -- PLE short-conv batched prefill pads every + # request in a batch to the batch-MAX query length, so one deep request inflates + # the whole step's activation above anything the profile saw. + # + # Raising this again requires re-running the deep-prefill probe in + # services/flash-next-mtp-bench/ and reading the allocator log, not just + # checking that the seat starts. Startup is not a depth test. + - --kv-cache-memory + - ${FN_KV_CACHE_MEMORY:-15032385536} # ⚠ 128K TO START, NOT the native 262,144 the checkpoint advertises, and the # gap is deliberate. Two open upstream issues make depth the risky axis here: # #54764 (PLE short-conv batched prefill pads every request to the batch-max diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index ca26bc5..e944b41 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -221,6 +221,46 @@ model_list: enable_thinking: true model_info: mode: chat + # gen-large -> Qwen3.8-Flash-Next ABLITERATED-NVFP4 (fv-ml1 GPU 2, :8022, + # flash-next-seat stack). Operator-requested test alias, added 2026-09-13. + # + # 176B total / ~6B active ultra-sparse MoE: a 125B main model plus a 51B n-gram (PLE) + # lookup table that lives in PINNED HOST RAM and is read by the GPU over CUDA UVA, so + # only ~74 GiB is resident on the card. 512 experts, 10 live per token; GDN linear + # attention on 36 of 48 layers, Qwen Sparse Attention on the other 12. + # + # SAMPLING IS THE CHECKPOINT'S OWN, not hand-tuned. generation_config.json declares + # temperature 1.0 / top_p 0.95 / top_k 20, and vLLM already applies them as the seat's + # defaults (it logs the override at boot). Restated here so a caller reading this file + # sees the EFFECTIVE values instead of inferring them. presence_penalty, min_p and + # repetition_penalty are deliberately UNSET — the checkpoint declares no canonical + # value for them, so none is invented. + # + # REASONING IS ON, at the seat's `medium` default. The Qwen3.8 chat template defaults + # to `xhigh`, where CoT length grows with conversation depth and has a long tail; the + # seat pins `medium` instead. Per-request chat_template_kwargs.reasoning_effort wins. + # + # ONE alias ON PURPOSE. The seat also serves a `-thinking` name, but a single alias + # cannot hit the shared-config enable_thinking mutation footgun -- that needs two + # aliases over the same (model, api_base) pair. If a thinking/non-thinking split is + # ever wanted, add gen-large-reasoning against the `-thinking` served name, the way + # gen / gen-reasoning are split above. + # + # NOT a speed upgrade over `gen` -- adopt for quality. Context is capped at 128K here, + # not the model's native 262K, and MTP speculative decoding is off pending a + # measurement on this hardware. Both are explained in + # stacks/flash-next-seat/README.md and services/flash-next-mtp-bench/README.md. + - model_name: gen-large + litellm_params: + model: hosted_vllm/qwen3.8-flash-next-uncensored + api_base: http://10.251.50.54:8022/v1 + api_key: os.environ/VLLM_API_KEY + temperature: 1.0 + top_p: 0.95 + extra_body: + top_k: 20 + model_info: + mode: chat # char-rp -> MeroMero-v2 NON-THINKING prose seat (:8016, vLLM, meromero-charrp stack on # ana-ml2 GPU 0). G4-MeroMero-v2-31B NVFP4A16, **Gemma-4 base** (google/gemma-4-31B-it), # 256K ctx, in-house quant. Replaced the GGUF/llama.cpp Magidonia-24B seat 2026-08-12.