flash-next-seat: Qwen3.8-Flash-Next on fv-ml1 GPU 2 with the n-gram table in host RAM
First seat whose weights do not fit its card. The model is 176B total -- a 125B main model plus a 51B n-gram (PLE) lookup table -- at ~6B active per token. The table is a pure embedding lookup, so it lives in pinned host RAM and the GPU reads rows directly over CUDA UVA: ~78 GiB resident on a 95.6 GiB card, 47.7 GiB pinned of 566 GB. GPU 2 and GPU 3 were both idle, so this displaced nothing. Checkpoint dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b99, pinned by revision: NVFP4 W4A4 routed experts, FP8 PLE table, everything else at source precision. Chosen over better-liked builds because its provenance states protocols and repeat counts -- AIME26 pass@1 98.75% over 30x8 repeats with a stated SEM, full-set GSM8K, and a byte-equality audit covering all 31 MTP tensors -- and because it declares text_config.ple_embedding_dtype, which is the field vLLM reads first when selecting the PLE weight format. Builds that ship an FP8 table without that declaration resolve to the unquantized path and fail on load; the README records the check. Requires vLLM #54371 (UVA PLE-offload, merged 2026-09-09T14:32Z), verified by ancestry: the pinned nightly is +150 commits / behind_by 0 from the merge commit. Not in v0.29.0, cut six hours earlier. The older worker-based offload (#53899) is paused upstream and is not the path here -- its deadlocks, ptrace gate and stale-output-under-graphs bugs all came from the separate worker process that UVA does not have. Five deliberate departures from the other seats on this box, each from a measurement rather than a preference, all annotated in place: - no MTP: the vLLM recipe measured it worse at every concurrency on 4xH100 (8-36% less throughput, 32-173% more latency, ~36% acceptance) - modelopt_fp4, not compressed-tensors: only the ModelOpt reader honours the ignore list keeping attention, shared experts, PLE and MTP out of W4A4 - KV left at auto: fp8 KV on this model's QSA path is an unmerged RFC (#54426) - mamba-cache-mode stated explicitly: the model raises on mode "all" - 128K context and 8192 batched tokens, not the native 262K: #54764 and #54919 make depth the risky axis, and sizing to the KV pool has never fixed a depth-driven crash on this hardware Nothing is wired into LiteLLM. Pointing an alias at this seat changes what existing callers receive and is a separate decision.
This commit is contained in:
@@ -0,0 +1,87 @@
|
||||
# flash-next-seat — Qwen3.8-Flash-Next (abliterated) on fv-ml1 GPU 2, :8022.
|
||||
# Copy to .env on the host at /opt/docker/compose/flash-next-seat/.env.
|
||||
#
|
||||
# This is the INITIAL configuration, stood up 2026-09-13. Values marked FIRST-BOOT
|
||||
# are deliberately conservative and expected to be revised once the seat has
|
||||
# reported its own memory budget and been bisected for depth. Do not treat them as
|
||||
# measured — they are not yet.
|
||||
|
||||
# ── Image ───────────────────────────────────────────────────────────────────
|
||||
# ⚠ MUST contain vLLM #54371 (UVA PLE-offload), merged 2026-09-09T14:32Z.
|
||||
# Verified by ancestry rather than version string: this commit is +150 / behind_by=0
|
||||
# from merge commit 3116c5d06bfe76501b3dd6b5434bfc7f3274f5e7. v0.29.0 does NOT
|
||||
# contain it (cut ~6h before the merge) and neither does any nightly-<sha> tag
|
||||
# dated 2026-09-09 or earlier — the nightly build runs ~06:16 UTC.
|
||||
FN_IMAGE=vllm/vllm-openai:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657
|
||||
API_KEY=replace-me
|
||||
|
||||
# ── Placement ───────────────────────────────────────────────────────────────
|
||||
# GPU 2 was completely idle (2 MiB) before this seat; GPU 3 still is. Every other
|
||||
# compose GPU pin on fv-ml1 is 0 or 1, so this seat displaced nothing.
|
||||
FN_GPU_ID=2
|
||||
FN_PORT=8022
|
||||
FN_CONTAINER_NAME=vllm-flash-next
|
||||
|
||||
# ── Model ───────────────────────────────────────────────────────────────────
|
||||
# dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b990578ef3031eccf9f28e675a289a09ee9
|
||||
FN_MODEL=/tank/aimodels/qwen38-flash-next-abliterated-nvfp4
|
||||
FN_QUANT=modelopt_fp4
|
||||
FN_SERVED_NAME=qwen3.8-flash-next-uncensored
|
||||
FN_SERVED_NAME_THINK=qwen3.8-flash-next-uncensored-thinking
|
||||
|
||||
# ── The offload ─────────────────────────────────────────────────────────────
|
||||
# The 51B n-gram table (47.7 GiB FP8) lives in pinned host RAM; the GPU reads rows
|
||||
# over CUDA UVA. Without this the checkpoint needs ~126 GiB of VRAM and will not
|
||||
# start on one 96 GiB card. fv-ml1 has 566 GB RAM / ~388 GB available, so the
|
||||
# host side is not a constraint here — unlike every DGX-Spark report upstream,
|
||||
# where host and device share one unified pool and "offload" frees nothing.
|
||||
FN_ENGRAM_CONFIG={"cpu_offload": true}
|
||||
|
||||
# ── Memory and context ──────────────────────────────────────────────────────
|
||||
# FIRST-BOOT. Card is dedicated, so 0.90 (the vLLM recipe's own figure for offload
|
||||
# configs) is not competing with a neighbour. Expected resident weights ~78 GiB of
|
||||
# 95.6, leaving ~8-9 GiB for KV + activation inside the 0.90 budget.
|
||||
# ⚠ NEXT STEP after first boot: read vLLM's budget line and replace this with an
|
||||
# explicit --kv-cache-memory in BYTES. A ratio sizes the KV calculation but does
|
||||
# not bound peak activation — that is what bit stacks/mog-sec twice on 2026-09-10.
|
||||
FN_GPU_MEM_UTIL=0.90
|
||||
|
||||
# FIRST-BOOT. 128K, against a 262,144 native ceiling. Raising this is gated on a
|
||||
# depth bisect with a NON-REPEATING prompt, because two open upstream issues make
|
||||
# depth the risky axis: #54764 (PLE short-conv prefill pads every request in a
|
||||
# batch to the batch-max query length) and #54919 (long prefill starving decode
|
||||
# for 3-7 minutes). See the compose comment.
|
||||
FN_MAX_MODEL_LEN=131072
|
||||
FN_MAX_NUM_SEQS=16
|
||||
FN_MAX_NUM_BATCHED_TOKENS=8192
|
||||
FN_MAMBA_CACHE_MODE=align
|
||||
|
||||
# ── Prefix caching ──────────────────────────────────────────────────────────
|
||||
# ⚠ THE ROLLBACK LEVER for open #54173 (CUBLAS_STATUS_INTERNAL_ERROR / illegal
|
||||
# memory access in the GDN path, WITH prefix caching). Set to the empty string to
|
||||
# disable. Leave FN_MAMBA_CACHE_MODE=align either way — Qwen4Exp raises on "all".
|
||||
FN_PREFIX_CACHING=--enable-prefix-caching
|
||||
|
||||
# ── Vision ──────────────────────────────────────────────────────────────────
|
||||
# 4194304 px = 2048x2048 -> ~5,125 image tokens. The checkpoint's own preprocessor
|
||||
# declares 16777216 (4096x4096) -> ~16,384 tokens, which is both wasteful and fatal
|
||||
# on builds enforcing the image-token count check. Same trap as stacks/mog-sec.
|
||||
FN_MM_PROCESSOR_KWARGS={"size": {"longest_edge": 4194304, "shortest_edge": 65536}}
|
||||
FN_LIMIT_MM={"image": 4}
|
||||
|
||||
# ── Misc ────────────────────────────────────────────────────────────────────
|
||||
FN_REASONING_PARSER=qwen3
|
||||
FN_REASONING_EFFORT=medium
|
||||
FN_TOOL_CALL_PARSER=qwen3_xml
|
||||
# Intentionally EMPTY. expandable_segments has corrupted retained tensors on this
|
||||
# box before (quant playbook §3.10) and has never been tested against a pinned
|
||||
# host allocation handed to UVA.
|
||||
FN_ALLOC_CONF=
|
||||
|
||||
# ── NOT SET, on purpose ─────────────────────────────────────────────────────
|
||||
# --speculative-config : MTP is off. vLLM's own recipe measured it WORSE at every
|
||||
# concurrency on 4xH100 (8-36% less throughput, 32-173%
|
||||
# more latency, ~36% acceptance) and open #55357 reports
|
||||
# episodic 0% acceptance with repetition collapse.
|
||||
# --kv-cache-dtype fp8 : fp8_e4m3 KV on this model's QSA path is an unmerged RFC
|
||||
# (#54426). Do not copy it over from gen/mog-sec.
|
||||
@@ -0,0 +1,139 @@
|
||||
# flash-next-seat — Qwen3.8-Flash-Next (abliterated), fv-ml1 GPU 2, `:8022`
|
||||
|
||||
The first seat on the fleet whose weights do not fit its card and run anyway.
|
||||
|
||||
`Qwen3.8-Flash-Next` is 176B total — a 125B main model plus a **51B n-gram (PLE)
|
||||
lookup table** — activating ~6B parameters per token. The n-gram table is a pure
|
||||
embedding lookup with almost no compute per token, so it lives in **pinned host
|
||||
RAM** and the GPU reads the rows it needs directly over **CUDA UVA** on a dedicated
|
||||
stream with async prefetch.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Checkpoint | `dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4` @ `be794b99…` (126.0 GiB) |
|
||||
| On the card | ~78 GiB of 95.6 GiB — routed experts NVFP4 W4A4, rest at source precision |
|
||||
| In host RAM | 47.7 GiB pinned, FP8 E4M3, 10 `model-plefp8-*` shards + per-table scalar scale |
|
||||
| Context | 131,072 to start (native ceiling 262,144) — see *Raising context* |
|
||||
| Speculative decoding | **none** — see *Why MTP is off* |
|
||||
| Gateway wiring | **none yet** — this seat is not in LiteLLM; `gen` is untouched |
|
||||
|
||||
## Deploy
|
||||
|
||||
```bash
|
||||
scripts/deploy-stack.sh fv-ml1 flash-next-seat # diffs vs live, prompts y/N
|
||||
# then on the host, first boot only:
|
||||
ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/flash-next-seat && docker compose config >/dev/null && docker compose up -d'
|
||||
```
|
||||
|
||||
The `.env` lives on the host and is never committed. Copy `.env.example`, set
|
||||
`API_KEY`, and read the FIRST-BOOT annotations before changing anything else.
|
||||
|
||||
## Architecture, briefly
|
||||
|
||||
Four ideas, and three of them shape the serving config:
|
||||
|
||||
- **GDN + QSA.** 36 of 48 layers use Gated DeltaNet (linear attention) to compress
|
||||
history; every fourth layer uses Qwen Sparse Attention for long-range retrieval.
|
||||
This is why KV is cheap at depth and why `--mamba-cache-dtype float32` matters.
|
||||
- **N-gram embedding.** The 51B lookup table that this seat offloads. Qwen's own
|
||||
framing: capacity with almost no per-token compute.
|
||||
- **Gated residual / hyper-connections.** Four residual branches; excluded from
|
||||
quantization in this checkpoint.
|
||||
- **MTP head.** Present and preserved byte-identically. Deliberately unused.
|
||||
|
||||
## Why this checkpoint, and the trap that disqualifies most others
|
||||
|
||||
vLLM selects the PLE table's weight format from **`text_config.ple_embedding_dtype`**,
|
||||
as the *first* branch of `Qwen4ExpPLEEmbeddingMethod.from_quant_config`. This
|
||||
checkpoint declares `"float8_e4m3fn"`.
|
||||
|
||||
A build that ships an FP8 PLE table **without** that declaration resolves instead
|
||||
through ModelOpt's `*.ple.*` exclude to the *unquantized* method, never registers
|
||||
the `weight_scale` parameter, and dies on load with `no module or parameter named
|
||||
'ngram_embedding.weight_scale'`. `gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8`
|
||||
is exactly this case. **Check that field before trying another build.**
|
||||
|
||||
Chosen over better-liked alternatives because its provenance states protocols and
|
||||
repeat counts rather than adjectives. From its own `qualification-notes.md` and
|
||||
metrics files, kept in the model directory:
|
||||
|
||||
- HarmBench, 240 genuinely-harmful behaviours, greedy: 100% compliance at reasoning
|
||||
off / low / xhigh.
|
||||
- MMLU 82.11% → 81.93% (−0.18 pp) on an identical harness, 2,280 questions.
|
||||
- GSM8K 97.27% (1283/1319), full set, single-shot, temp 0.6 — inside the stated
|
||||
BF16 reference band 97.12–97.50.
|
||||
- AIME26 pass@1 98.75% (237/240, **SEM 0.61 pp, 30 problems × 8 repeats**),
|
||||
majority@8 100%, `max_tokens` 130,000, 4.9M completion tokens, stop_rate 99.17%.
|
||||
- Byte-equality audit of unchanged tensors: 1,562 tensors / 118.4 GB compared, all
|
||||
passed, **including all 31 MTP tensors**.
|
||||
|
||||
⚠ Two honest gaps in that evidence. The routed experts are **NVFP4 W4A4**, and
|
||||
nobody — including the publisher — has measured this checkpoint at the full 262K
|
||||
context; AIME26's 130K-token generations are the deepest evidence that exists.
|
||||
Separately, `validate_checkpoint_report.json` in the repo describes the *earlier
|
||||
BF16-PLE revision* (204 shards / 173.6 GiB), not the published FP8-PLE one.
|
||||
|
||||
Rejected alternatives, for the record:
|
||||
`orcarouter/…-Uncensored-NVFP4` is gated (access request pending nothing — not
|
||||
requested); `nvidia/…-NVFP4` is the cleanest ModelOpt MIXED_PRECISION build but is
|
||||
not abliterated; `lovedheart/…-Pruned-RTXPRO-6000` prunes to 448 of 512 experts.
|
||||
|
||||
## Why MTP is off
|
||||
|
||||
Against our house graft-MTP habit, and on purpose.
|
||||
|
||||
vLLM's own recipe for this model measured MTP on 4×H100 as **worse at every
|
||||
concurrency tested** — 8–36% lower request throughput, 32–173% higher per-token
|
||||
latency, driven by ~36% acceptance — and says do not enable it by default. Open
|
||||
issue **#55357** reports episodic 0% draft acceptance with repetition collapse
|
||||
inside thinking blocks. Open **#55496** reports ModelOpt `MIXED_PRECISION` failing
|
||||
to load FP8_BLOCK_SCALES MTP experts.
|
||||
|
||||
Turning it on is two lines in `compose.yaml` (documented in place). If you do,
|
||||
measure it **here**, with repeats, against this seat's own baseline — the numbers
|
||||
above are someone else's hardware.
|
||||
|
||||
## The upstream situation, as of 2026-09-13
|
||||
|
||||
- **#53896** — model support. **Merged 2026-08-31.** In v0.29.0.
|
||||
- **#54371** — *UVA PLE-offload and Engram tensor parallelism*. **Merged
|
||||
2026-09-09T14:32Z.** This is the offload this seat uses. **Not in v0.29.0**,
|
||||
which was cut ~6 h earlier; present in `v0.29.1rc0` and in any nightly from
|
||||
2026-09-10 onward.
|
||||
- **#53899** — the *older, worker-based* PLE offload. **Open and explicitly paused**
|
||||
in favour of #54371. Do not go back to it. Its whole bug family — the TP=1
|
||||
startup deadlock (#53960), the `pidfd_getfd` / `kernel.yama.ptrace_scope` gate,
|
||||
the shared-CUDA-event race under async scheduling, and silently one-step-stale
|
||||
PLE outputs under CUDA graphs — came from the separate worker process and the
|
||||
CUDA-IPC row transfer that the UVA path does not have.
|
||||
|
||||
Open issues worth knowing about on SM120, none of them blocking:
|
||||
|
||||
| Issue | What it does | Our exposure |
|
||||
|---|---|---|
|
||||
| **#54173** | CUBLAS internal error / illegal memory access in the GDN path **with prefix caching** | We enable prefix caching. `FN_PREFIX_CACHING=` is the one-line rollback. |
|
||||
| **#54764** | PLE short-conv batched prefill pads every request to the batch-max query length | Why `--max-num-batched-tokens` is 8192, not 16384 |
|
||||
| **#54919** | Long prefill starves active decode for 3–7 minutes | Why context starts at 128K |
|
||||
| **#54521** | Greedy decoding non-deterministic from `persistent_topk` in prefill | Affects any A/B on this seat — establish a noise floor before comparing |
|
||||
| **#54426** | fp8_e4m3 KV on the QSA path is an unmerged RFC | Why `--kv-cache-dtype` is **not** set to fp8 here |
|
||||
|
||||
## Raising context
|
||||
|
||||
128K is a starting value, not a measured one. Before raising it, bisect with a
|
||||
**non-repeating** prompt — a repeated one hashes to cached blocks and never
|
||||
prefills deep, so it proves nothing. The `stacks/mog-sec` README records this the
|
||||
hard way: three successive context cuts all sized the *KV pool* while the crashes
|
||||
were governed by *processing depth*, which is a different number.
|
||||
|
||||
The point of a ceiling is the refusal. Below it the seat serves; above it vLLM
|
||||
returns a clean 400 naming the limit, instead of the engine dying and taking every
|
||||
in-flight request with it.
|
||||
|
||||
## Not done yet
|
||||
|
||||
- **Pin `--kv-cache-memory` in bytes** from the first boot's budget line, replacing
|
||||
the 0.90 ratio. Same discipline as `stacks/mog-sec` and `stacks/erp-seat`.
|
||||
- **Gateway wiring is deliberately absent.** Pointing any LiteLLM alias at this
|
||||
seat — in particular displacing `gen` / `summarizer` / `classifier`, which is the
|
||||
long-term intent recorded in henge item 49 — changes what every existing caller
|
||||
receives and is the operator's call, not a deploy-time default.
|
||||
@@ -0,0 +1,212 @@
|
||||
# flash-next-seat — Qwen3.8-Flash-Next (abliterated) on fv-ml1 GPU 2 (:8022).
|
||||
#
|
||||
# THE POINT OF THIS SEAT: it is the first model on the fleet whose weights do NOT
|
||||
# fit the card, and it runs anyway. Qwen3.8-Flash-Next is 176B total — a 125B main
|
||||
# model plus a 51B n-gram (PLE) lookup table — at ~6B active per token. The n-gram
|
||||
# table is a pure lookup with almost no compute, so it lives in PINNED HOST RAM and
|
||||
# the GPU reads the rows it needs directly over CUDA UVA. Resident on the card:
|
||||
# ~78 GiB of 95.6. Resident in host RAM: 47.7 GiB of 566 GB. See --engram-config.
|
||||
#
|
||||
# Serves dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4, revision
|
||||
# be794b990578ef3031eccf9f28e675a289a09ee9, pulled to
|
||||
# /tank/aimodels/qwen38-flash-next-abliterated-nvfp4 (126.0 GiB on disk).
|
||||
# - routed experts (294,912 tensors, 48 layers x 512 experts) NVFP4 W4A4
|
||||
# - PLE n-gram table FP8 E4M3, 10 `model-plefp8-*` shards + per-table scalar scale,
|
||||
# taken from Qwen/Qwen3.8-Flash-Next-FP8 rather than re-derived
|
||||
# - everything else source precision: self_attn, linear_attn, mlp.gate,
|
||||
# shared_expert, hyper_connection, ple projections, visual tower, embeddings,
|
||||
# lm_head and all 31 MTP tensors are in the quant config's `ignore` list
|
||||
# - abliteration is a direct weight edit, no fine-tune: HarmBench 240-real-harm
|
||||
# compliance 100% at reasoning off/low/xhigh, MMLU 82.11 -> 81.93 (-0.18pp),
|
||||
# GSM8K 97.27% (full 1319), AIME26 pass@1 98.75% (30 x 8 repeats, SEM 0.61pp).
|
||||
# Publisher's own reports are kept in the model dir; they state protocols and
|
||||
# repeat counts, which is why this build was chosen over better-liked ones.
|
||||
#
|
||||
# ⚠ THE CHECKPOINT-SELECTION TRAP, recorded because it silently disqualifies most
|
||||
# community builds. vLLM picks the PLE table's format from
|
||||
# `text_config.ple_embedding_dtype` as the FIRST branch of
|
||||
# Qwen4ExpPLEEmbeddingMethod.from_quant_config. This checkpoint declares
|
||||
# "float8_e4m3fn". A checkpoint with an FP8 PLE table and NO such declaration
|
||||
# (e.g. gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8) resolves instead
|
||||
# through ModelOpt's `*.ple.*` exclude to the UNQUANTIZED method and fails to load
|
||||
# on the scale tensor it does ship. Check that field before trying another build.
|
||||
#
|
||||
# ⚠ IMAGE: must contain vLLM PR #54371 ("UVA PLE-offload and Engram tensor
|
||||
# parallelism", merged 2026-09-09T14:32Z). Verified by ancestry, not version string:
|
||||
# nightly-eed1f3d0... is +150 commits / behind_by=0 from merge commit 3116c5d0.
|
||||
# NOT in v0.29.0 — that release was cut ~6h before the merge. The older worker-based
|
||||
# offload (#53899) is PAUSED upstream in favour of this one; do not go back to it,
|
||||
# its entire bug family (TP=1 deadlock, pidfd_getfd/yama ptrace gate, shared-event
|
||||
# race, silently one-step-stale outputs under CUDA graphs) came from the separate
|
||||
# worker process that the UVA path does not have.
|
||||
#
|
||||
# ⚠⚠ MTP IS DELIBERATELY OFF, against our house graft-MTP habit. The model ships a
|
||||
# working MTP head and this checkpoint preserves it byte-identically (31 tensors,
|
||||
# audit passed) — we still do not enable it. vLLM's own recipe measured MTP on
|
||||
# 4xH100 as WORSE AT EVERY CONCURRENCY tested: 8-36% lower request throughput,
|
||||
# 32-173% higher per-token latency, ~36% acceptance, and says do not default it on.
|
||||
# Open #55357 reports episodic 0% draft acceptance with repetition collapse inside
|
||||
# thinking blocks. If you want it, it is FN_SPEC_CONFIG — but measure it here, with
|
||||
# repeats, before believing it.
|
||||
name: flash-next-seat
|
||||
services:
|
||||
vllm-flash-next:
|
||||
image: ${FN_IMAGE:-vllm/vllm-openai:latest}
|
||||
container_name: ${FN_CONTAINER_NAME:-vllm-flash-next}
|
||||
restart: unless-stopped
|
||||
ipc: host
|
||||
# The PLE table is 47.7 GiB of PINNED host memory. Pinned pages are not
|
||||
# swappable and count against the process's locked-memory limit, so lift it.
|
||||
ulimits:
|
||||
memlock: -1
|
||||
ports:
|
||||
- "${FN_PORT:-8022}:8000"
|
||||
volumes:
|
||||
- /tank/aimodels/huggingface:/hfcache
|
||||
- ${FN_MODEL:-/tank/aimodels/qwen38-flash-next-abliterated-nvfp4}:/model:ro
|
||||
environment:
|
||||
- HF_HOME=/hfcache
|
||||
- HF_HUB_CACHE=/hfcache/hub
|
||||
- VLLM_API_KEY=${API_KEY:-}
|
||||
# Intentionally UNSET by default. expandable_segments has corrupted retained
|
||||
# tensors elsewhere on this box (quant playbook §3.10) and the offload path
|
||||
# hands UVA a pinned host allocation — not a combination anyone has tested.
|
||||
- PYTORCH_CUDA_ALLOC_CONF=${FN_ALLOC_CONF:-}
|
||||
command:
|
||||
- /model
|
||||
- --served-model-name
|
||||
- ${FN_SERVED_NAME:-qwen3.8-flash-next-uncensored}
|
||||
- ${FN_SERVED_NAME_THINK:-qwen3.8-flash-next-uncensored-thinking}
|
||||
- --host
|
||||
- 0.0.0.0
|
||||
- --port
|
||||
- "8000"
|
||||
# ⚠ modelopt_fp4, NOT compressed-tensors as on the other seats here. This
|
||||
# checkpoint's quantization_config carries compressed-tensors-shaped
|
||||
# `config_groups` AND `quant_method: modelopt` / `quant_algo: NVFP4`
|
||||
# (producer nvidia-modelopt 0.46.0). The ModelOpt reader is the one whose
|
||||
# per-prefix `is_layer_excluded` honours the 13-entry ignore list that keeps
|
||||
# attention, the shared experts, the PLE and MTP out of W4A4.
|
||||
- --quantization
|
||||
- ${FN_QUANT:-modelopt_fp4}
|
||||
# ── THE WHOLE REASON THIS SEAT FITS ──────────────────────────────────────
|
||||
# Keeps the 51B n-gram table in pinned host RAM, read by the GPU over UVA on
|
||||
# a dedicated stream with async prefetch. Legacy VLLM_PLE_CPU_OFFLOAD=1 still
|
||||
# works but is deprecated in favour of this flag.
|
||||
# `embedding_across_dp` is irrelevant at TP=1/DP=1 — left at its default.
|
||||
- --engram-config
|
||||
- '${FN_ENGRAM_CONFIG:-{"cpu_offload": true}}'
|
||||
# GPU 2 is DEDICATED to this seat — nothing else is resident on the card, so
|
||||
# unlike gen/mog-sec on GPU 0/1 this ratio is not competing with a neighbour.
|
||||
# 0.90 is the vLLM recipe's own figure for offload configurations.
|
||||
# ⚠ PIN --kv-cache-memory IN BYTES once the first boot reports its budget
|
||||
# line. Same lesson as stacks/mog-sec and stacks/erp-seat: a ratio sizes the
|
||||
# KV calculation but does not bound peak activation, and it silently yields a
|
||||
# different cache depending on what is resident at start time.
|
||||
- --gpu-memory-utilization
|
||||
- ${FN_GPU_MEM_UTIL:-0.90}
|
||||
# ⚠ 128K TO START, NOT the native 262,144 the checkpoint advertises, and the
|
||||
# gap is deliberate. Two open upstream issues make depth the risky axis here:
|
||||
# #54764 (PLE short-conv batched prefill pads every request to the batch-max
|
||||
# query length, so one deep request inflates the whole step's activation) and
|
||||
# #54919 (long prefill starving active decode for 3-7 minutes). The mog-sec
|
||||
# lesson applies directly: what the KV pool can HOLD and what the card can
|
||||
# PROCESS at depth are different numbers, and sizing to the pool never fixed
|
||||
# a depth-driven crash. RAISE THIS ONLY AFTER BISECTING with a NON-REPEATING
|
||||
# prompt — a repeated one hashes to cached blocks and never prefills deep.
|
||||
- --max-model-len
|
||||
- ${FN_MAX_MODEL_LEN:-131072}
|
||||
- --max-num-seqs
|
||||
- ${FN_MAX_NUM_SEQS:-16}
|
||||
# 8192 rather than the 16384 used elsewhere, for the #54764 padding reason
|
||||
# above: a chunk's activation cost here scales with the batch's DEEPEST
|
||||
# request, not its average. Costs prefill throughput, costs no context.
|
||||
- --max-num-batched-tokens
|
||||
- ${FN_MAX_NUM_BATCHED_TOKENS:-8192}
|
||||
- --trust-remote-code
|
||||
- --dtype
|
||||
- auto
|
||||
# Hybrid backbone: 36 of 48 layers are Gated DeltaNet (linear attention), the
|
||||
# other 12 are Qwen Sparse Attention. The model's own config asks for
|
||||
# mamba_ssm_dtype float32.
|
||||
- --mamba-cache-dtype
|
||||
- float32
|
||||
# ⚠ MANDATORY-ISH: Qwen4Exp RAISES NotImplementedError on mamba_cache_mode
|
||||
# "all" and tells you to use "align". "align" is already the implicit default
|
||||
# when prefix caching is on, but it is stated here so that turning prefix
|
||||
# caching off cannot silently select a mode the model rejects.
|
||||
- --mamba-cache-mode
|
||||
- ${FN_MAMBA_CACHE_MODE:-align}
|
||||
# ⚠ FIRST THING TO TURN OFF if you see CUBLAS_STATUS_INTERNAL_ERROR or an
|
||||
# illegal memory access: open #54173 is exactly that, in the GDN path, WITH
|
||||
# prefix caching. On by default because every vLLM recipe config for this
|
||||
# model enables it.
|
||||
- ${FN_PREFIX_CACHING:---enable-prefix-caching}
|
||||
- --enable-chunked-prefill
|
||||
# ⚠ KV CACHE LEFT AT AUTO — do NOT copy `--kv-cache-dtype fp8` from the gen or
|
||||
# mog-sec seats. fp8_e4m3 KV on this model's QSA path is an unmerged RFC
|
||||
# (#54426: working patch, one machine, author asking for corroboration).
|
||||
#
|
||||
# Recipe sets this in every configuration; autotune has no value here and
|
||||
# costs startup time.
|
||||
- --no-enable-flashinfer-autotune
|
||||
- --limit-mm-per-prompt
|
||||
- '${FN_LIMIT_MM:-{"image": 4}}'
|
||||
# ⚠ MANDATORY. This checkpoint's preprocessor_config.json declares
|
||||
# size.longest_edge = 16777216 px (4096x4096), which expands to ~16,384 image
|
||||
# tokens — one image eating 12% of a 128K context, and enough to fail startup
|
||||
# on builds that enforce the text-vs-ids token count check. Same trap already
|
||||
# documented on stacks/mog-sec. 4194304 = 2048x2048 -> ~5,125 tokens.
|
||||
- --mm-processor-kwargs
|
||||
- '${FN_MM_PROCESSOR_KWARGS:-{"size": {"longest_edge": 4194304, "shortest_edge": 65536}}}'
|
||||
- --reasoning-parser
|
||||
- ${FN_REASONING_PARSER:-qwen3}
|
||||
# The Qwen3.8 chat template defaults to `xhigh`, where reasoning length grows
|
||||
# with conversation depth and has a long tail — measured on the 27B sibling at
|
||||
# 4,529-5,532 chars spiking to 9,261. `medium` is the seat-level default;
|
||||
# per-request override via chat_template_kwargs still works.
|
||||
- --default-chat-template-kwargs
|
||||
- '{"reasoning_effort": "${FN_REASONING_EFFORT:-medium}"}'
|
||||
- --enable-auto-tool-choice
|
||||
# qwen3_xml, per the vLLM recipe for THIS model — not the qwen3_coder parser
|
||||
# the mog-sec seat uses.
|
||||
- --tool-call-parser
|
||||
- ${FN_TOOL_CALL_PARSER:-qwen3_xml}
|
||||
# ── NO --speculative-config LINE, ON PURPOSE ─────────────────────────────
|
||||
# Speculative decoding is absent rather than disabled-by-a-flag, because
|
||||
# there is no honest "off" value for --speculative-config and a placeholder
|
||||
# flag in this slot is the kind of thing that gets copied forward and
|
||||
# misread. To turn MTP on, add these two lines here and nothing else:
|
||||
# - --speculative-config
|
||||
# - '{"method": "mtp", "num_speculative_tokens": 3}'
|
||||
# Read the MTP warning in the header first; the measured default is off.
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids:
|
||||
- "${FN_GPU_ID:-2}"
|
||||
capabilities:
|
||||
- gpu
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
# ⚠ LONG. Cold boot reads 126 GiB from /tank and pins 47.7 GiB of it; the
|
||||
# closest published single-card timing is ~8.5 min load on a 96 GB card.
|
||||
# 1800s so a slow first boot is not mistaken for a failed one.
|
||||
start_period: 1800s
|
||||
retries: 3
|
||||
networks:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=AI - Inference
|
||||
- homepage.name=Qwen3.8-Flash-Next (uncensored)
|
||||
- homepage.icon=mdi-lightning-bolt-outline
|
||||
- homepage.description=176B/6B-active MoE, abliterated, NVFP4 + 48GB n-gram table offloaded to host RAM — fv-ml1 GPU 2
|
||||
- homepage.href=http://10.251.50.54:${FN_PORT:-8022}/docs
|
||||
networks:
|
||||
tnet:
|
||||
name: traefik-net
|
||||
external: true
|
||||
Reference in New Issue
Block a user