vh
7e62a07341
Operator-directed: raise context to the model's native maximum and take as much KV as the card safely allows, and expose the seat through LiteLLM as `gen-large`. max_model_len 131,072 -> 262,144 KV cache 8.76 -> 14.00 GiB (332,721 -> 560,654 tokens) concurrency 2.54x@128K -> 2.14x@262K ⚠ 16.00 GiB WAS TRIED FIRST AND IS TOO AGGRESSIVE. A 155,497-token non-repeating prefill drove GPU 2 to 97,074 of 97,887 MiB and the caching allocator logged "OOM on device 0 while trying to allocate 488636416 bytes (free: 422117376)" -- 466 MiB wanted against 403 MiB free. The request completed, so nothing failed visibly; that is one step before the shape that crashed stacks/mog-sec twice on 2026-09-10 (~1.04 GiB wanted, ~600 MB free). Backed off to 14.00 GiB, which re-probes clean: zero allocator warnings, a 155,557-token prefill in 14.2 s, and 2,085 MiB still free at peak. The reason the first estimate was wrong is worth keeping, because it is not obvious and it inverts the usual advice: --kv-cache-memory makes vLLM SKIP MEMORY PROFILING ENTIRELY and ignore --gpu-memory-utilization. The profiler was the thing accounting for deep-prefill activation, so pinning bytes switched off the protection that the pin was supposed to formalise. vLLM's own "--kv-cache-memory=18745235968 (17.46 GiB) to fully utilize gpu memory" line is computed from a profile measured at max-num-batched-tokens depth and sits 3.5 GiB above what a 150K-token request survives; open #54764 compounds it, since PLE short-conv prefill pads every request in a batch to the batch-MAX query length. max-num-batched-tokens stays at 8192 -- it is what bounds the activation peak, and doubling max_model_len left the profiled peak unchanged at 1.65 GiB precisely because the peak tracks chunk size, not context length. Gateway: `gen-large` added to the LiteLLM model_list, pointing at fv-ml1:8022. One alias on purpose -- a single alias cannot trip the shared-config enable_thinking mutation footgun, which needs two over the same (model, api_base). Sampling is the checkpoint's own declared set (temp 1.0 / top_p 0.95 / top_k 20); presence_penalty, min_p and repetition_penalty are left unset because the checkpoint declares no canonical value for them. Verified registered for both the infra-ops admin key and the shared all-agents key, since a new model behind a scoped allowlist 403s silently. Also adds services/flash-next-mtp-bench/ -- the MTP measurement campaign and its rationale. MTP stays off, but on "not yet measured here" rather than on vLLM's 4xH100 recipe number, which is a cross-harness comparison and not evidence about a TP=1 Blackwell seat.
262 lines
15 KiB
YAML
262 lines
15 KiB
YAML
# flash-next-seat — Qwen3.8-Flash-Next (abliterated) on fv-ml1 GPU 2 (:8022).
|
|
#
|
|
# THE POINT OF THIS SEAT: it is the first model on the fleet whose weights do NOT
|
|
# fit the card, and it runs anyway. Qwen3.8-Flash-Next is 176B total — a 125B main
|
|
# model plus a 51B n-gram (PLE) lookup table — at ~6B active per token. The n-gram
|
|
# table is a pure lookup with almost no compute, so it lives in PINNED HOST RAM and
|
|
# the GPU reads the rows it needs directly over CUDA UVA. Resident on the card:
|
|
# ~78 GiB of 95.6. Resident in host RAM: 47.7 GiB of 566 GB. See --engram-config.
|
|
#
|
|
# Serves dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4, revision
|
|
# be794b990578ef3031eccf9f28e675a289a09ee9, pulled to
|
|
# /tank/aimodels/qwen38-flash-next-abliterated-nvfp4 (126.0 GiB on disk).
|
|
# - routed experts (294,912 tensors, 48 layers x 512 experts) NVFP4 W4A4
|
|
# - PLE n-gram table FP8 E4M3, 10 `model-plefp8-*` shards + per-table scalar scale,
|
|
# taken from Qwen/Qwen3.8-Flash-Next-FP8 rather than re-derived
|
|
# - everything else source precision: self_attn, linear_attn, mlp.gate,
|
|
# shared_expert, hyper_connection, ple projections, visual tower, embeddings,
|
|
# lm_head and all 31 MTP tensors are in the quant config's `ignore` list
|
|
# - abliteration is a direct weight edit, no fine-tune: HarmBench 240-real-harm
|
|
# compliance 100% at reasoning off/low/xhigh, MMLU 82.11 -> 81.93 (-0.18pp),
|
|
# GSM8K 97.27% (full 1319), AIME26 pass@1 98.75% (30 x 8 repeats, SEM 0.61pp).
|
|
# Publisher's own reports are kept in the model dir; they state protocols and
|
|
# repeat counts, which is why this build was chosen over better-liked ones.
|
|
#
|
|
# ⚠ THE CHECKPOINT-SELECTION TRAP, recorded because it silently disqualifies most
|
|
# community builds. vLLM picks the PLE table's format from
|
|
# `text_config.ple_embedding_dtype` as the FIRST branch of
|
|
# Qwen4ExpPLEEmbeddingMethod.from_quant_config. This checkpoint declares
|
|
# "float8_e4m3fn". A checkpoint with an FP8 PLE table and NO such declaration
|
|
# (e.g. gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8) resolves instead
|
|
# through ModelOpt's `*.ple.*` exclude to the UNQUANTIZED method and fails to load
|
|
# on the scale tensor it does ship. Check that field before trying another build.
|
|
#
|
|
# ⚠ IMAGE: must contain vLLM PR #54371 ("UVA PLE-offload and Engram tensor
|
|
# parallelism", merged 2026-09-09T14:32Z). Verified by ancestry, not version string:
|
|
# nightly-eed1f3d0... is +150 commits / behind_by=0 from merge commit 3116c5d0.
|
|
# NOT in v0.29.0 — that release was cut ~6h before the merge. The older worker-based
|
|
# offload (#53899) is PAUSED upstream in favour of this one; do not go back to it,
|
|
# its entire bug family (TP=1 deadlock, pidfd_getfd/yama ptrace gate, shared-event
|
|
# race, silently one-step-stale outputs under CUDA graphs) came from the separate
|
|
# worker process that the UVA path does not have.
|
|
#
|
|
# ⚠⚠ MTP IS DELIBERATELY OFF, against our house graft-MTP habit. The model ships a
|
|
# working MTP head and this checkpoint preserves it byte-identically (31 tensors,
|
|
# audit passed) — we still do not enable it. vLLM's own recipe measured MTP on
|
|
# 4xH100 as WORSE AT EVERY CONCURRENCY tested: 8-36% lower request throughput,
|
|
# 32-173% higher per-token latency, ~36% acceptance, and says do not default it on.
|
|
# Open #55357 reports episodic 0% draft acceptance with repetition collapse inside
|
|
# thinking blocks. If you want it, it is FN_SPEC_CONFIG — but measure it here, with
|
|
# repeats, before believing it.
|
|
name: flash-next-seat
|
|
services:
|
|
vllm-flash-next:
|
|
image: ${FN_IMAGE:-vllm/vllm-openai:latest}
|
|
container_name: ${FN_CONTAINER_NAME:-vllm-flash-next}
|
|
restart: unless-stopped
|
|
ipc: host
|
|
# The PLE table is 47.7 GiB of PINNED host memory. Pinned pages are not
|
|
# swappable and count against the process's locked-memory limit, so lift it.
|
|
ulimits:
|
|
memlock: -1
|
|
ports:
|
|
- "${FN_PORT:-8022}:8000"
|
|
volumes:
|
|
- /tank/aimodels/huggingface:/hfcache
|
|
- ${FN_MODEL:-/tank/aimodels/qwen38-flash-next-abliterated-nvfp4}:/model:ro
|
|
environment:
|
|
- HF_HOME=/hfcache
|
|
- HF_HUB_CACHE=/hfcache/hub
|
|
- VLLM_API_KEY=${API_KEY:-}
|
|
# Intentionally UNSET by default. expandable_segments has corrupted retained
|
|
# tensors elsewhere on this box (quant playbook §3.10) and the offload path
|
|
# hands UVA a pinned host allocation — not a combination anyone has tested.
|
|
- PYTORCH_CUDA_ALLOC_CONF=${FN_ALLOC_CONF:-}
|
|
command:
|
|
- /model
|
|
- --served-model-name
|
|
- ${FN_SERVED_NAME:-qwen3.8-flash-next-uncensored}
|
|
- ${FN_SERVED_NAME_THINK:-qwen3.8-flash-next-uncensored-thinking}
|
|
- --host
|
|
- 0.0.0.0
|
|
- --port
|
|
- "8000"
|
|
# ⚠ modelopt_fp4, NOT compressed-tensors as on the other seats here. This
|
|
# checkpoint's quantization_config carries compressed-tensors-shaped
|
|
# `config_groups` AND `quant_method: modelopt` / `quant_algo: NVFP4`
|
|
# (producer nvidia-modelopt 0.46.0). The ModelOpt reader is the one whose
|
|
# per-prefix `is_layer_excluded` honours the 13-entry ignore list that keeps
|
|
# attention, the shared experts, the PLE and MTP out of W4A4.
|
|
- --quantization
|
|
- ${FN_QUANT:-modelopt_fp4}
|
|
# ── THE WHOLE REASON THIS SEAT FITS ──────────────────────────────────────
|
|
# Keeps the 51B n-gram table in pinned host RAM, read by the GPU over UVA on
|
|
# a dedicated stream with async prefetch. Legacy VLLM_PLE_CPU_OFFLOAD=1 still
|
|
# works but is deprecated in favour of this flag.
|
|
# `embedding_across_dp` is irrelevant at TP=1/DP=1 — left at its default.
|
|
- --engram-config
|
|
- '${FN_ENGRAM_CONFIG:-{"cpu_offload": true}}'
|
|
# GPU 2 is DEDICATED to this seat — nothing else is resident on the card, so
|
|
# unlike gen/mog-sec on GPU 0/1 this ratio is not competing with a neighbour.
|
|
# 0.90 is the vLLM recipe's own figure for offload configurations.
|
|
# ⚠ PIN --kv-cache-memory IN BYTES once the first boot reports its budget
|
|
# line. Same lesson as stacks/mog-sec and stacks/erp-seat: a ratio sizes the
|
|
# KV calculation but does not bound peak activation, and it silently yields a
|
|
# different cache depending on what is resident at start time.
|
|
- --gpu-memory-utilization
|
|
- ${FN_GPU_MEM_UTIL:-0.96}
|
|
# ⚠ KV PINNED IN BYTES, and the figure is MEASURED at 262K, not derived.
|
|
# The ratio above sizes the KV calculation but does NOT bound actual usage:
|
|
# peak activation is profiled once at startup and real long-context work
|
|
# exceeds the profile. stacks/mog-sec proved that twice on 2026-09-10, where
|
|
# vLLM's own budget line showed the seat running 0.9 GiB OVER its stated
|
|
# reservation. An explicit figure is reproducible; a ratio silently yields a
|
|
# different cache depending on what else is resident at start time.
|
|
#
|
|
# ⚠⚠ 16.00 GiB WAS TRIED FIRST AND WAS TOO AGGRESSIVE -- MEASURED, 2026-09-13.
|
|
# A 155,497-token non-repeating prefill drove GPU 2 to 97,074 of 97,887 MiB and
|
|
# the allocator logged
|
|
# OOM on device 0 while trying to allocate 488636416 bytes (free: 422117376)
|
|
# i.e. it wanted 466 MiB with 403 MiB left. The request still completed (the
|
|
# caching allocator recovered) but that is ONE STEP before the mog-sec crash
|
|
# shape: ~1.04 GiB wanted with ~600 MB free. 14.00 GiB restores ~2 GiB of
|
|
# headroom and was chosen over the 16.00 that nearly failed.
|
|
#
|
|
# ⚠ THE REASON THE FIRST ESTIMATE WAS WRONG, because it is not obvious: setting
|
|
# --kv-cache-memory makes vLLM SKIP MEMORY PROFILING ENTIRELY and ignore
|
|
# --gpu-memory-utilization ("reserved 16.0 GiB ... and skipped memory profiling.
|
|
# This does not respect the gpu_memory_utilization config"). So the 0.96 cap
|
|
# stops protecting anything, and the 1.65 GiB "peak activation" figure -- which
|
|
# was profiled against max-num-batched-tokens=8192, not against real depth --
|
|
# stops being subtracted at all. The profiler was the thing keeping deep
|
|
# prefills safe, and pinning bytes turns it off.
|
|
#
|
|
# 15,032,385,536 B = 14.00 GiB. For reference the 262K boot's own accounting:
|
|
# free on device 94.42 GiB
|
|
# weights+non-torch 75.07
|
|
# peak activation 1.65 (profiled against max-num-batched-tokens,
|
|
# 8192 here -- NOT against max-model-len,
|
|
# which is why doubling context to 262K left
|
|
# this number unchanged)
|
|
# CUDA graphs 0.10
|
|
# → theoretical KV 17.60 GiB (on paper; NOT achievable at depth)
|
|
#
|
|
# ⚠ vLLM's own "--kv-cache-memory=18745235968 (17.46 GiB) to fully utilize gpu
|
|
# memory" suggestion is 3.5 GiB ABOVE what survives a deep prefill here. Do not
|
|
# take it. That line is computed from the startup profile, which is measured at
|
|
# max-num-batched-tokens depth and says nothing about a 150K-token request.
|
|
# Open upstream #54764 compounds it -- PLE short-conv batched prefill pads every
|
|
# request in a batch to the batch-MAX query length, so one deep request inflates
|
|
# the whole step's activation above anything the profile saw.
|
|
#
|
|
# Raising this again requires re-running the deep-prefill probe in
|
|
# services/flash-next-mtp-bench/ and reading the allocator log, not just
|
|
# checking that the seat starts. Startup is not a depth test.
|
|
- --kv-cache-memory
|
|
- ${FN_KV_CACHE_MEMORY:-15032385536}
|
|
# ⚠ 128K TO START, NOT the native 262,144 the checkpoint advertises, and the
|
|
# gap is deliberate. Two open upstream issues make depth the risky axis here:
|
|
# #54764 (PLE short-conv batched prefill pads every request to the batch-max
|
|
# query length, so one deep request inflates the whole step's activation) and
|
|
# #54919 (long prefill starving active decode for 3-7 minutes). The mog-sec
|
|
# lesson applies directly: what the KV pool can HOLD and what the card can
|
|
# PROCESS at depth are different numbers, and sizing to the pool never fixed
|
|
# a depth-driven crash. RAISE THIS ONLY AFTER BISECTING with a NON-REPEATING
|
|
# prompt — a repeated one hashes to cached blocks and never prefills deep.
|
|
- --max-model-len
|
|
- ${FN_MAX_MODEL_LEN:-131072}
|
|
- --max-num-seqs
|
|
- ${FN_MAX_NUM_SEQS:-16}
|
|
# 8192 rather than the 16384 used elsewhere, for the #54764 padding reason
|
|
# above: a chunk's activation cost here scales with the batch's DEEPEST
|
|
# request, not its average. Costs prefill throughput, costs no context.
|
|
- --max-num-batched-tokens
|
|
- ${FN_MAX_NUM_BATCHED_TOKENS:-8192}
|
|
- --trust-remote-code
|
|
- --dtype
|
|
- auto
|
|
# Hybrid backbone: 36 of 48 layers are Gated DeltaNet (linear attention), the
|
|
# other 12 are Qwen Sparse Attention. The model's own config asks for
|
|
# mamba_ssm_dtype float32.
|
|
- --mamba-cache-dtype
|
|
- float32
|
|
# ⚠ MANDATORY-ISH: Qwen4Exp RAISES NotImplementedError on mamba_cache_mode
|
|
# "all" and tells you to use "align". "align" is already the implicit default
|
|
# when prefix caching is on, but it is stated here so that turning prefix
|
|
# caching off cannot silently select a mode the model rejects.
|
|
- --mamba-cache-mode
|
|
- ${FN_MAMBA_CACHE_MODE:-align}
|
|
# ⚠ FIRST THING TO TURN OFF if you see CUBLAS_STATUS_INTERNAL_ERROR or an
|
|
# illegal memory access: open #54173 is exactly that, in the GDN path, WITH
|
|
# prefix caching. On by default because every vLLM recipe config for this
|
|
# model enables it.
|
|
- ${FN_PREFIX_CACHING:---enable-prefix-caching}
|
|
- --enable-chunked-prefill
|
|
# ⚠ KV CACHE LEFT AT AUTO — do NOT copy `--kv-cache-dtype fp8` from the gen or
|
|
# mog-sec seats. fp8_e4m3 KV on this model's QSA path is an unmerged RFC
|
|
# (#54426: working patch, one machine, author asking for corroboration).
|
|
#
|
|
# Recipe sets this in every configuration; autotune has no value here and
|
|
# costs startup time.
|
|
- --no-enable-flashinfer-autotune
|
|
- --limit-mm-per-prompt
|
|
- '${FN_LIMIT_MM:-{"image": 4}}'
|
|
# ⚠ MANDATORY. This checkpoint's preprocessor_config.json declares
|
|
# size.longest_edge = 16777216 px (4096x4096), which expands to ~16,384 image
|
|
# tokens — one image eating 12% of a 128K context, and enough to fail startup
|
|
# on builds that enforce the text-vs-ids token count check. Same trap already
|
|
# documented on stacks/mog-sec. 4194304 = 2048x2048 -> ~5,125 tokens.
|
|
- --mm-processor-kwargs
|
|
- '${FN_MM_PROCESSOR_KWARGS:-{"size": {"longest_edge": 4194304, "shortest_edge": 65536}}}'
|
|
- --reasoning-parser
|
|
- ${FN_REASONING_PARSER:-qwen3}
|
|
# The Qwen3.8 chat template defaults to `xhigh`, where reasoning length grows
|
|
# with conversation depth and has a long tail — measured on the 27B sibling at
|
|
# 4,529-5,532 chars spiking to 9,261. `medium` is the seat-level default;
|
|
# per-request override via chat_template_kwargs still works.
|
|
- --default-chat-template-kwargs
|
|
- '{"reasoning_effort": "${FN_REASONING_EFFORT:-medium}"}'
|
|
- --enable-auto-tool-choice
|
|
# qwen3_xml, per the vLLM recipe for THIS model — not the qwen3_coder parser
|
|
# the mog-sec seat uses.
|
|
- --tool-call-parser
|
|
- ${FN_TOOL_CALL_PARSER:-qwen3_xml}
|
|
# ── NO --speculative-config LINE, ON PURPOSE ─────────────────────────────
|
|
# Speculative decoding is absent rather than disabled-by-a-flag, because
|
|
# there is no honest "off" value for --speculative-config and a placeholder
|
|
# flag in this slot is the kind of thing that gets copied forward and
|
|
# misread. To turn MTP on, add these two lines here and nothing else:
|
|
# - --speculative-config
|
|
# - '{"method": "mtp", "num_speculative_tokens": 3}'
|
|
# Read the MTP warning in the header first; the measured default is off.
|
|
deploy:
|
|
resources:
|
|
reservations:
|
|
devices:
|
|
- driver: nvidia
|
|
device_ids:
|
|
- "${FN_GPU_ID:-2}"
|
|
capabilities:
|
|
- gpu
|
|
healthcheck:
|
|
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
|
interval: 30s
|
|
timeout: 10s
|
|
# ⚠ LONG. Cold boot reads 126 GiB from /tank and pins 47.7 GiB of it; the
|
|
# closest published single-card timing is ~8.5 min load on a 96 GB card.
|
|
# 1800s so a slow first boot is not mistaken for a failed one.
|
|
start_period: 1800s
|
|
retries: 3
|
|
networks:
|
|
- tnet
|
|
labels:
|
|
- homepage.group=AI - Inference
|
|
- homepage.name=Qwen3.8-Flash-Next (uncensored)
|
|
- homepage.icon=mdi-lightning-bolt-outline
|
|
- homepage.description=176B/6B-active MoE, abliterated, NVFP4 + 48GB n-gram table offloaded to host RAM — fv-ml1 GPU 2
|
|
- homepage.href=http://10.251.50.54:${FN_PORT:-8022}/docs
|
|
networks:
|
|
tnet:
|
|
name: traefik-net
|
|
external: true
|