vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

flash-next-seat: full 262K context, KV pinned at a measured 14 GiB, gen-large on the gateway

Browse Source 
Operator-directed: raise context to the model's native maximum and take as much KV
as the card safely allows, and expose the seat through LiteLLM as `gen-large`.

  max_model_len     131,072  ->  262,144
  KV cache             8.76  ->  14.00 GiB  (332,721 -> 560,654 tokens)
  concurrency      2.54x@128K ->  2.14x@262K

⚠ 16.00 GiB WAS TRIED FIRST AND IS TOO AGGRESSIVE. A 155,497-token non-repeating
prefill drove GPU 2 to 97,074 of 97,887 MiB and the caching allocator logged "OOM on
device 0 while trying to allocate 488636416 bytes (free: 422117376)" -- 466 MiB wanted
against 403 MiB free. The request completed, so nothing failed visibly; that is one
step before the shape that crashed stacks/mog-sec twice on 2026-09-10 (~1.04 GiB
wanted, ~600 MB free). Backed off to 14.00 GiB, which re-probes clean: zero allocator
warnings, a 155,557-token prefill in 14.2 s, and 2,085 MiB still free at peak.

The reason the first estimate was wrong is worth keeping, because it is not obvious
and it inverts the usual advice: --kv-cache-memory makes vLLM SKIP MEMORY PROFILING
ENTIRELY and ignore --gpu-memory-utilization. The profiler was the thing accounting
for deep-prefill activation, so pinning bytes switched off the protection that the
pin was supposed to formalise. vLLM's own "--kv-cache-memory=18745235968 (17.46 GiB)
to fully utilize gpu memory" line is computed from a profile measured at
max-num-batched-tokens depth and sits 3.5 GiB above what a 150K-token request
survives; open #54764 compounds it, since PLE short-conv prefill pads every request
in a batch to the batch-MAX query length.

max-num-batched-tokens stays at 8192 -- it is what bounds the activation peak, and
doubling max_model_len left the profiled peak unchanged at 1.65 GiB precisely because
the peak tracks chunk size, not context length.

Gateway: `gen-large` added to the LiteLLM model_list, pointing at fv-ml1:8022. One
alias on purpose -- a single alias cannot trip the shared-config enable_thinking
mutation footgun, which needs two over the same (model, api_base). Sampling is the
checkpoint's own declared set (temp 1.0 / top_p 0.95 / top_k 20); presence_penalty,
min_p and repetition_penalty are left unset because the checkpoint declares no
canonical value for them. Verified registered for both the infra-ops admin key and
the shared all-agents key, since a new model behind a scoped allowlist 403s silently.

Also adds services/flash-next-mtp-bench/ -- the MTP measurement campaign and its
rationale. MTP stays off, but on "not yet measured here" rather than on vLLM's
4xH100 recipe number, which is a cross-harness comparison and not evidence about a
TP=1 Blackwell seat.
This commit is contained in:
Vuong Hoang
2026-09-12 23:49:06 -07:00
parent 3132a16ca0
commit 7e62a07341
5 changed files with 387 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
services/flash-next-mtp-bench/README.md
+106
View File
@@ -0,0 +1,106 @@
# Does MTP actually hurt Qwen3.8-Flash-Next on one Blackwell card?
Status: **campaign built, running.** Results and a verdict land here with the raw
JSON alongside, so the conclusion can be re-derived rather than taken on faith.
## Why this exists
The `flash-next-seat` stack shipped with MTP speculative decoding **off**, citing
vLLM's published recipe: on 4×H100 it measured MTP as worse at every concurrency
tested — 8–36% lower request throughput, 32–173% higher per-token latency, ~36%
acceptance — and says don't default it on.
**That was not valid evidence about our seat, and defaulting on it was the wrong
call.** Our own measurement rule says the harness is part of the number and that
cross-harness comparisons are invalid, not merely noisy. The recipe's harness
differs from ours on nearly every axis that could plausibly drive the result:
| | vLLM recipe | this seat |
|---|---|---|
| GPUs | 4× H100 80 GB (Hopper) | 1× RTX PRO 6000 96 GB (Blackwell, sm_120) |
| Parallelism | TP=4 — experts sharded 4 ways, all-reduce per layer | **TP=1 — every expert local, no collective** |
| PLE table | offloaded (forced: 80 GB cards can't hold it) | offloaded (chosen) |
| k tested | **3 only** | 1, 2, 3 |
## Two mechanisms that could make MTP lose here, and one that makes k the real question
Spec decoding's usual win is that verifying k+1 tokens costs about the same as
decoding 1, because decode is memory-bandwidth-bound: you read the weights once
either way. **Two things about this model break that assumption.**
1. **Sparse-MoE expert-read amplification.** 512 experts, 10 active per token. At
batch 1, decoding one token touches ~10 experts. Verifying k+1 tokens routes
each position to *its own* 10, largely disjoint — so the weights read scale
with the token count instead of staying flat. The "free verification" premise
does not hold for an ultra-sparse MoE. This is worst at low concurrency and
shrinks as the batch already touches many experts anyway, which is exactly why
the sweep has to cross concurrency and not just report one number.
2. **PLE/UVA fetch amplification.** The 51B n-gram table lives in host RAM and is
read over PCIe. Every token position needs its own row lookups, so k+1 tokens
means k+1× the host round-trips — multiplying traffic on the single slowest
link in the system, the one we deliberately moved off the card.
**And the mechanism that makes k the actual experiment:** the checkpoint's
`mtp_num_hidden_layers` is **1**. The draft head is a *single module run
autoregressively* for k>1 — the pattern our own quant playbook §5.1 flags, where
deeper k improves acceptance and destroys throughput. If throughput falls
monotonically in k while acceptance rises, then **k=1 may win and the recipe's
k=3 number says nothing about it.** Nobody has published k=1 for this model.
So there are coherent reasons MTP could genuinely lose here — and an equally
coherent reason the published number is the wrong number to decide on.
## Design
Five boots. Arms in order: `off_A`, `k1`, `k2`, `k3`, `off_B`.
- **Repeats** — 3 per (arm, concurrency) cell, concurrency ∈ {1, 4, 8}. Median
reported with spread, never a single run.
- **Noise floor** — `off_A` and `off_B` are the *same configuration*, booted first
and last. Their difference is the floor, and because they are separate boots it
includes boot-to-boot variance that three reps inside one boot cannot see.
Running `off_B` last also catches monotonic drift across the campaign.
- **Positive control (a)** — MTP acceptance must be **> 0** on every MTP arm, read
from `/metrics` as a delta. A head that loads uninitialised serves fine and
reports ~0% accept; that is the failure our `gen` seat's `re:^mtp.*`
ignore-list footgun produces. **An arm with ~0% acceptance is void** — it
measured a broken head, not MTP. (Pre-checked: this checkpoint excludes
`mtp.*` and `model.mtp.*` from quantization in both quant configs, and its own
audit reports all 31 MTP tensors byte-identical to source.)
- **Positive control (b)** — aggregate throughput must **rise with concurrency**
inside every arm. Known-true for a batching server; if the harness can't see
it, the harness is blind and its negatives are worthless.
- **Null control** — `off_A` vs `off_B` must show no effect beyond the floor.
- **Stated sensitivity floor** — the report prints "cannot resolve effects smaller
than X" from the observed `off_A`/`off_B` spread. A delta under it is not a
finding.
**Harness parity.** Every arm's argv is *derived from the live compose file* via
`docker compose config`, not retyped, so the arms are provably identical except
for `--speculative-config`. That is what stops a stray flag from quietly becoming
the real independent variable.
## Instruments
- `concbench.py` — the **house** concurrency harness, reused unchanged from
`services/gen-seat-mixed-quant/bench/`. It already avoids the three traps that
have produced confident wrong answers here before: unseeded nonces so prefix
caching can't fake prefill, MTP acceptance read as a **delta** so a long-lived
seat's history doesn't swamp it, and aggregate throughput from wall clock rather
than a sum of per-request rates.
- `run-campaign.sh` — the driver. Runs on fv-ml1 against `127.0.0.1` so the
NH3↔FV mesh hop is not in the measurement.
## Known gaps in this design
- **No TTFT.** `concbench.py` doesn't stream, so prefill latency isn't measured
separately. The MTP mechanisms above are decode-side, so this is an acceptable
omission — but it means the campaign cannot speak to first-token latency.
- **One prompt shape.** ~400 output tokens from a short prompt. MTP acceptance is
workload-dependent; a long-context or code workload could differ and this says
nothing about them.
- **The offload's contribution is not isolated.** If MTP loses, this design cannot
separate mechanism 1 from mechanism 2. The clean discriminator is available and
cheap — re-run the best MTP arm at TP=2 across GPU 2+3 with `cpu_offload: false`
so the table is resident, and compare the MTP delta with and without the PCIe
path. GPU 3 is idle, so this is a follow-up worth doing if the answer matters.
services/flash-next-mtp-bench/run-campaign.sh
Executable
+165
View File
@@ -0,0 +1,165 @@
#!/usr/bin/env bash
# run-campaign.sh — does MTP speculative decoding help or hurt Qwen3.8-Flash-Next
# on ONE RTX PRO 6000 with the n-gram table offloaded to host RAM?
#
# WHY THIS EXISTS. vLLM's published recipe measured MTP on 4xH100 as worse at
# every concurrency (8-36% less throughput, 32-173% more per-token latency, ~36%
# acceptance) and we initially defaulted MTP off on that basis. That was a
# cross-harness comparison and therefore not valid evidence about THIS seat:
# 4xH100 is TP=4 Hopper with experts sharded four ways and an all-reduce per
# layer; this is TP=1 Blackwell with every expert local. Our own rule says the
# harness is part of the number and cross-harness comparisons are invalid, not
# merely noisy. So we measure it here.
#
# AND THE RECIPE ONLY TESTED k=3. The checkpoint's mtp_num_hidden_layers is 1, so
# the draft head is a SINGLE module run autoregressively for k>1 (quant playbook
# §5.1: deeper k improves acceptance and destroys throughput). If throughput
# falls monotonically in k while acceptance rises, k=1 may well WIN and the
# recipe's k=3 number says nothing about it. That is the hypothesis this sweeps.
#
# ── MEASUREMENT DISCIPLINE ──────────────────────────────────────────────────
# Every number here carries repeats, a noise floor, a positive control and a null
# control, or it does not get to carry a conclusion.
#
# REPEATS 3 reps per (arm, concurrency) cell; median reported with spread.
# NOISE FLOOR The `off` arm is booted TWICE -- off_A first and off_B last. Both
# arms are the same configuration, so their difference IS the
# floor, and it includes boot-to-boot variance, which three reps
# inside one boot cannot see. Running off_B last also catches any
# monotonic drift across the campaign (thermals, page cache).
# POSITIVE CTL (a) MTP acceptance must be > 0 on every MTP arm. A head that
# loads uninitialised reports ~0% accept and still serves --
# the exact failure our gen seat's `re:^mtp.*` ignore-list
# footgun produces. If acceptance is ~0 the throughput number
# is measuring a broken head, not MTP, and the arm is void.
# (b) Aggregate throughput must RISE with concurrency inside every
# arm. That is known-true for a batching server; if the
# harness cannot see it, the harness is blind and its nulls
# are worthless.
# NULL CONTROL off_A vs off_B must show no effect beyond the floor.
# FLOOR STATED The report prints "cannot resolve effects smaller than X" from
# the observed off_A/off_B spread. A delta under it is not a
# finding.
#
# HARNESS PARITY. Every arm's argv is DERIVED from the live compose file, so the
# arms are provably identical except for --speculative-config. Nothing is
# retyped, which is what stops a stray flag from becoming the real independent
# variable.
#
# The production container is stopped for the duration: same GPU, same port. The
# seat has no consumers yet, so this costs nothing. It is restored at the end.
set -uo pipefail
STACK_DIR=${STACK_DIR:-/opt/docker/compose/flash-next-seat}
OUT=${OUT:-/tank/aimodels/flash-next-mtp-bench}
BENCH=${BENCH:-$OUT/concbench.py}
PORT=${PORT:-8022}
GPU=${GPU:-2}
REPS=${REPS:-3}
CONCS=${CONCS:-"1 4 8"}
MAXTOK=${MAXTOK:-400}
RPS=${RPS:-4} # requests per stream, per concbench
METHOD=${METHOD:-qwen4_exp_mtp}
MODEL=${MODEL:-qwen3.8-flash-next-uncensored}
NAME=fn-mtp-bench
BOOT_TIMEOUT=${BOOT_TIMEOUT:-2400}
mkdir -p "$OUT"
log(){ echo "[$(date -u +%H:%M:%S)] $*" | tee -a "$OUT/campaign.log"; }
# --- derive the production argv + run opts straight from the compose file -----
read_compose() {
sudo -n docker compose -f "$STACK_DIR/compose.yaml" --env-file "$STACK_DIR/.env" config --format json
}
log "deriving argv from $STACK_DIR/compose.yaml"
read_compose > "$OUT/resolved-compose.json" || { log "FATAL: could not resolve compose"; exit 1; }
python3 - "$OUT/resolved-compose.json" "$OUT/argv.txt" "$OUT/envs.txt" <<'PY'
import json,sys
d=json.load(open(sys.argv[1]))
svc=d["services"]["vllm-flash-next"]
open(sys.argv[2],"w").write("\n".join(str(x) for x in svc["command"])+"\n")
env=svc.get("environment") or {}
if isinstance(env,dict): items=[(k,v) for k,v in env.items()]
else: items=[e.split("=",1) for e in env]
open(sys.argv[3],"w").write("\n".join(f"{k}={'' if v is None else v}" for k,v in items)+"\n")
PY
mapfile -t ARGV < "$OUT/argv.txt"
ENVARGS=(); while IFS= read -r l; do [ -n "$l" ] && ENVARGS+=(-e "$l"); done < "$OUT/envs.txt"
log "argv has ${#ARGV[@]} entries; ${#ENVARGS[@]} env flags"
printf '%s\n' "${ARGV[@]}" | sed 's/^/ /' >> "$OUT/campaign.log"
MODEL_DIR=$(python3 -c "
import json,sys
d=json.load(open('$OUT/resolved-compose.json'))
for v in d['services']['vllm-flash-next']['volumes']:
t=v['target'] if isinstance(v,dict) else v.split(':')[1]
s=v['source'] if isinstance(v,dict) else v.split(':')[0]
if t=='/model': print(s)
")
HFCACHE=$(python3 -c "
import json
d=json.load(open('$OUT/resolved-compose.json'))
for v in d['services']['vllm-flash-next']['volumes']:
t=v['target'] if isinstance(v,dict) else v.split(':')[1]
s=v['source'] if isinstance(v,dict) else v.split(':')[0]
if t=='/hfcache': print(s)
")
IMAGE=$(python3 -c "
import json; print(json.load(open('$OUT/resolved-compose.json'))['services']['vllm-flash-next']['image'])")
log "image=$IMAGE model=$MODEL_DIR"
stop_bench(){ sudo -n docker rm -f "$NAME" >/dev/null 2>&1 || true; }
trap 'stop_bench' EXIT
boot(){ # boot <arm-label> [spec-json]
local arm="$1"; shift
local extra=(); [ $# -gt 0 ] && [ -n "${1:-}" ] && extra=(--speculative-config "$1")
stop_bench
log "=== boot arm=$arm ${extra[*]:-(no spec-config)}"
sudo -n docker run -d --name "$NAME" --ipc host --ulimit memlock=-1 \
--gpus "\"device=$GPU\"" -p "$PORT:8000" \
-v "$HFCACHE:/hfcache" -v "$MODEL_DIR:/model:ro" \
"${ENVARGS[@]}" "$IMAGE" "${ARGV[@]}" "${extra[@]}" \
> "$OUT/$arm.cid" 2>"$OUT/$arm.runerr" || { log " docker run FAILED: $(cat "$OUT/$arm.runerr")"; return 1; }
local t0=$SECONDS
while [ $((SECONDS-t0)) -lt "$BOOT_TIMEOUT" ]; do
if curl -sf "http://127.0.0.1:$PORT/health" >/dev/null 2>&1; then
log " healthy after $((SECONDS-t0))s"
sudo -n docker logs "$NAME" > "$OUT/$arm.boot.log" 2>&1
return 0
fi
if ! sudo -n docker ps --format '{{.Names}}' | grep -qx "$NAME"; then
log " CONTAINER DIED during boot"; sudo -n docker logs "$NAME" > "$OUT/$arm.boot.log" 2>&1
tail -25 "$OUT/$arm.boot.log" | sed 's/^/ /' | tee -a "$OUT/campaign.log"; return 1
fi
sleep 15
done
log " BOOT TIMEOUT after ${BOOT_TIMEOUT}s"; sudo -n docker logs "$NAME" > "$OUT/$arm.boot.log" 2>&1; return 1
}
bench_arm(){ # bench_arm <arm-label>
local arm="$1" cargs=()
for c in $CONCS; do cargs+=(--concurrency "$c"); done
for rep in $(seq 1 "$REPS"); do
log " bench $arm rep=$rep"
python3 "$BENCH" --base "http://127.0.0.1:$PORT" --model "$MODEL" \
"${cargs[@]}" --requests-per-stream "$RPS" --max-tokens "$MAXTOK" \
--tag "$arm-rep$rep" --out "$OUT/res-$arm-rep$rep.json" 2>&1 | tee -a "$OUT/campaign.log"
done
}
# --- the campaign -----------------------------------------------------------
# off_A first, the MTP arms in ascending k, off_B LAST so the floor brackets the
# whole run rather than sitting at one end of it.
run_arm(){ boot "$1" "${2:-}" && bench_arm "$1" || log " arm $1 SKIPPED (boot failed)"; }
log "########## CAMPAIGN START (reps=$REPS concs='$CONCS' max_tokens=$MAXTOK) ##########"
run_arm off_A ""
run_arm k1 "{\"method\":\"$METHOD\",\"num_speculative_tokens\":1}"
run_arm k2 "{\"method\":\"$METHOD\",\"num_speculative_tokens\":2}"
run_arm k3 "{\"method\":\"$METHOD\",\"num_speculative_tokens\":3}"
run_arm off_B ""
stop_bench
log "########## CAMPAIGN DONE -- results in $OUT ##########"
log "restoring the production container"
sudo -n docker compose -f "$STACK_DIR/compose.yaml" --env-file "$STACK_DIR/.env" up -d 2>&1 | tail -3 | tee -a "$OUT/campaign.log"
stacks/flash-next-seat/.env.example
+26 -14
View File
@@ -37,21 +37,33 @@ FN_SERVED_NAME_THINK=qwen3.8-flash-next-uncensored-thinking
# where host and device share one unified pool and "offload" frees nothing.
FN_ENGRAM_CONFIG={"cpu_offload": true}
# ── Memory and context ──────────────────────────────────────────────────────
# FIRST-BOOT. Card is dedicated, so 0.90 (the vLLM recipe's own figure for offload
# configs) is not competing with a neighbour. Expected resident weights ~78 GiB of
# 95.6, leaving ~8-9 GiB for KV + activation inside the 0.90 budget.
# ⚠ NEXT STEP after first boot: read vLLM's budget line and replace this with an
# explicit --kv-cache-memory in BYTES. A ratio sizes the KV calculation but does
# not bound peak activation — that is what bit stacks/mog-sec twice on 2026-09-10.
FN_GPU_MEM_UTIL=0.90
# Card is dedicated -- one tenant, nothing to compete with. Measured at 262K:
# weights 74.36 GiB resident, 560,654 KV tokens, 2.14x concurrency.
# ⚠ This ratio is now ADVISORY ONLY -- see the KV pin below, which overrides it.
FN_GPU_MEM_UTIL=0.96
# ⚠⚠ 14.00 GiB, PINNED IN BYTES AND MEASURED THE HARD WAY (2026-09-13).
# 16.00 GiB was tried first and nearly OOM'd: a 155,497-token prefill drove GPU 2 to
# 97,074 of 97,887 MiB and the allocator logged "OOM on device 0 while trying to
# allocate 488636416 bytes (free: 422117376)" -- 466 MiB wanted, 403 MiB free. The
# request survived but that is one step before the mog-sec crash shape.
# ⚠ WHY THE ESTIMATE WAS WRONG: --kv-cache-memory makes vLLM SKIP MEMORY PROFILING
# and ignore --gpu-memory-utilization entirely. The profiler was the thing accounting
# for deep-prefill activation; pinning bytes turns it off. Do NOT take vLLM's
# "17.46 GiB to fully utilize" suggestion -- it is computed from a profile taken at
# max-num-batched-tokens depth and is 3.5 GiB above what a 150K request survives.
# Re-raising requires re-running the deep probe and reading the allocator log.
# VERIFIED at 14.00 GiB: 0 OOM warnings, 155,557-token prefill in 14.2 s, 2,085 MiB
# still free on the card at peak.
FN_KV_CACHE_MEMORY=15032385536
# FIRST-BOOT. 128K, against a 262,144 native ceiling. Raising this is gated on a
# depth bisect with a NON-REPEATING prompt, because two open upstream issues make
# depth the risky axis: #54764 (PLE short-conv prefill pads every request in a
# batch to the batch-max query length) and #54919 (long prefill starving decode
# for 3-7 minutes). See the compose comment.
FN_MAX_MODEL_LEN=131072
# FULL NATIVE 262,144 (operator-directed 2026-09-13). The KV pool holds ~641K
# tokens, so a single max-length request fits with ~2.4x concurrency to spare.
# ⚠ STARTUP IS NOT A DEPTH TEST. Two open upstream issues make depth the risky
# axis -- #54764 (PLE short-conv prefill pads every request in a batch to the
# batch-MAX query length) and #54919 (long prefill starving active decode for 3-7
# minutes) -- and vLLM's own recipe admits a single 262K request was never tested.
# If deep requests misbehave, --max-num-batched-tokens is the lever, not this.
FN_MAX_MODEL_LEN=262144
FN_MAX_NUM_SEQS=16
FN_MAX_NUM_BATCHED_TOKENS=8192
FN_MAMBA_CACHE_MODE=align
stacks/flash-next-seat/compose.yaml
+50 -1
View File
@@ -104,7 +104,56 @@ services:
# KV calculation but does not bound peak activation, and it silently yields a
# different cache depending on what is resident at start time.
- --gpu-memory-utilization
- ${FN_GPU_MEM_UTIL:-0.90}
- ${FN_GPU_MEM_UTIL:-0.96}
# ⚠ KV PINNED IN BYTES, and the figure is MEASURED at 262K, not derived.
# The ratio above sizes the KV calculation but does NOT bound actual usage:
# peak activation is profiled once at startup and real long-context work
# exceeds the profile. stacks/mog-sec proved that twice on 2026-09-10, where
# vLLM's own budget line showed the seat running 0.9 GiB OVER its stated
# reservation. An explicit figure is reproducible; a ratio silently yields a
# different cache depending on what else is resident at start time.
#
# ⚠⚠ 16.00 GiB WAS TRIED FIRST AND WAS TOO AGGRESSIVE -- MEASURED, 2026-09-13.
# A 155,497-token non-repeating prefill drove GPU 2 to 97,074 of 97,887 MiB and
# the allocator logged
# OOM on device 0 while trying to allocate 488636416 bytes (free: 422117376)
# i.e. it wanted 466 MiB with 403 MiB left. The request still completed (the
# caching allocator recovered) but that is ONE STEP before the mog-sec crash
# shape: ~1.04 GiB wanted with ~600 MB free. 14.00 GiB restores ~2 GiB of
# headroom and was chosen over the 16.00 that nearly failed.
#
# ⚠ THE REASON THE FIRST ESTIMATE WAS WRONG, because it is not obvious: setting
# --kv-cache-memory makes vLLM SKIP MEMORY PROFILING ENTIRELY and ignore
# --gpu-memory-utilization ("reserved 16.0 GiB ... and skipped memory profiling.
# This does not respect the gpu_memory_utilization config"). So the 0.96 cap
# stops protecting anything, and the 1.65 GiB "peak activation" figure -- which
# was profiled against max-num-batched-tokens=8192, not against real depth --
# stops being subtracted at all. The profiler was the thing keeping deep
# prefills safe, and pinning bytes turns it off.
#
# 15,032,385,536 B = 14.00 GiB. For reference the 262K boot's own accounting:
# free on device 94.42 GiB
# weights+non-torch 75.07
# peak activation 1.65 (profiled against max-num-batched-tokens,
# 8192 here -- NOT against max-model-len,
# which is why doubling context to 262K left
# this number unchanged)
# CUDA graphs 0.10
# → theoretical KV 17.60 GiB (on paper; NOT achievable at depth)
#
# ⚠ vLLM's own "--kv-cache-memory=18745235968 (17.46 GiB) to fully utilize gpu
# memory" suggestion is 3.5 GiB ABOVE what survives a deep prefill here. Do not
# take it. That line is computed from the startup profile, which is measured at
# max-num-batched-tokens depth and says nothing about a 150K-token request.
# Open upstream #54764 compounds it -- PLE short-conv batched prefill pads every
# request in a batch to the batch-MAX query length, so one deep request inflates
# the whole step's activation above anything the profile saw.
#
# Raising this again requires re-running the deep-prefill probe in
# services/flash-next-mtp-bench/ and reading the allocator log, not just
# checking that the seat starts. Startup is not a depth test.
- --kv-cache-memory
- ${FN_KV_CACHE_MEMORY:-15032385536}
# ⚠ 128K TO START, NOT the native 262,144 the checkpoint advertises, and the
# gap is deliberate. Two open upstream issues make depth the risky axis here:
# #54764 (PLE short-conv batched prefill pads every request to the batch-max
stacks/litellm/conf/config.yaml
+40
View File
@@ -221,6 +221,46 @@ model_list:
enable_thinking: true
model_info:
mode: chat
# gen-large -> Qwen3.8-Flash-Next ABLITERATED-NVFP4 (fv-ml1 GPU 2, :8022,
# flash-next-seat stack). Operator-requested test alias, added 2026-09-13.
#
# 176B total / ~6B active ultra-sparse MoE: a 125B main model plus a 51B n-gram (PLE)
# lookup table that lives in PINNED HOST RAM and is read by the GPU over CUDA UVA, so
# only ~74 GiB is resident on the card. 512 experts, 10 live per token; GDN linear
# attention on 36 of 48 layers, Qwen Sparse Attention on the other 12.
#
# SAMPLING IS THE CHECKPOINT'S OWN, not hand-tuned. generation_config.json declares
# temperature 1.0 / top_p 0.95 / top_k 20, and vLLM already applies them as the seat's
# defaults (it logs the override at boot). Restated here so a caller reading this file
# sees the EFFECTIVE values instead of inferring them. presence_penalty, min_p and
# repetition_penalty are deliberately UNSET — the checkpoint declares no canonical
# value for them, so none is invented.
#
# REASONING IS ON, at the seat's `medium` default. The Qwen3.8 chat template defaults
# to `xhigh`, where CoT length grows with conversation depth and has a long tail; the
# seat pins `medium` instead. Per-request chat_template_kwargs.reasoning_effort wins.
#
# ONE alias ON PURPOSE. The seat also serves a `-thinking` name, but a single alias
# cannot hit the shared-config enable_thinking mutation footgun -- that needs two
# aliases over the same (model, api_base) pair. If a thinking/non-thinking split is
# ever wanted, add gen-large-reasoning against the `-thinking` served name, the way
# gen / gen-reasoning are split above.
#
# NOT a speed upgrade over `gen` -- adopt for quality. Context is capped at 128K here,
# not the model's native 262K, and MTP speculative decoding is off pending a
# measurement on this hardware. Both are explained in
# stacks/flash-next-seat/README.md and services/flash-next-mtp-bench/README.md.
- model_name: gen-large
litellm_params:
model: hosted_vllm/qwen3.8-flash-next-uncensored
api_base: http://10.251.50.54:8022/v1
api_key: os.environ/VLLM_API_KEY
temperature: 1.0
top_p: 0.95
extra_body:
top_k: 20
model_info:
mode: chat
# char-rp -> MeroMero-v2 NON-THINKING prose seat (:8016, vLLM, meromero-charrp stack on
# ana-ml2 GPU 0). G4-MeroMero-v2-31B NVFP4A16, **Gemma-4 base** (google/gemma-4-31B-it),
# 256K ctx, in-house quant. Replaced the GGUF/llama.cpp Magidonia-24B seat 2026-08-12.