vh
f964a47262
First seat whose weights do not fit its card. The model is 176B total -- a 125B main model plus a 51B n-gram (PLE) lookup table -- at ~6B active per token. The table is a pure embedding lookup, so it lives in pinned host RAM and the GPU reads rows directly over CUDA UVA: ~78 GiB resident on a 95.6 GiB card, 47.7 GiB pinned of 566 GB. GPU 2 and GPU 3 were both idle, so this displaced nothing. Checkpoint dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b99, pinned by revision: NVFP4 W4A4 routed experts, FP8 PLE table, everything else at source precision. Chosen over better-liked builds because its provenance states protocols and repeat counts -- AIME26 pass@1 98.75% over 30x8 repeats with a stated SEM, full-set GSM8K, and a byte-equality audit covering all 31 MTP tensors -- and because it declares text_config.ple_embedding_dtype, which is the field vLLM reads first when selecting the PLE weight format. Builds that ship an FP8 table without that declaration resolve to the unquantized path and fail on load; the README records the check. Requires vLLM #54371 (UVA PLE-offload, merged 2026-09-09T14:32Z), verified by ancestry: the pinned nightly is +150 commits / behind_by 0 from the merge commit. Not in v0.29.0, cut six hours earlier. The older worker-based offload (#53899) is paused upstream and is not the path here -- its deadlocks, ptrace gate and stale-output-under-graphs bugs all came from the separate worker process that UVA does not have. Five deliberate departures from the other seats on this box, each from a measurement rather than a preference, all annotated in place: - no MTP: the vLLM recipe measured it worse at every concurrency on 4xH100 (8-36% less throughput, 32-173% more latency, ~36% acceptance) - modelopt_fp4, not compressed-tensors: only the ModelOpt reader honours the ignore list keeping attention, shared experts, PLE and MTP out of W4A4 - KV left at auto: fp8 KV on this model's QSA path is an unmerged RFC (#54426) - mamba-cache-mode stated explicitly: the model raises on mode "all" - 128K context and 8192 batched tokens, not the native 262K: #54764 and #54919 make depth the risky axis, and sizing to the KV pool has never fixed a depth-driven crash on this hardware Nothing is wired into LiteLLM. Pointing an alias at this seat changes what existing callers receive and is a separate decision.
88 lines
6.0 KiB
Bash
88 lines
6.0 KiB
Bash
# flash-next-seat — Qwen3.8-Flash-Next (abliterated) on fv-ml1 GPU 2, :8022.
|
|
# Copy to .env on the host at /opt/docker/compose/flash-next-seat/.env.
|
|
#
|
|
# This is the INITIAL configuration, stood up 2026-09-13. Values marked FIRST-BOOT
|
|
# are deliberately conservative and expected to be revised once the seat has
|
|
# reported its own memory budget and been bisected for depth. Do not treat them as
|
|
# measured — they are not yet.
|
|
|
|
# ── Image ───────────────────────────────────────────────────────────────────
|
|
# ⚠ MUST contain vLLM #54371 (UVA PLE-offload), merged 2026-09-09T14:32Z.
|
|
# Verified by ancestry rather than version string: this commit is +150 / behind_by=0
|
|
# from merge commit 3116c5d06bfe76501b3dd6b5434bfc7f3274f5e7. v0.29.0 does NOT
|
|
# contain it (cut ~6h before the merge) and neither does any nightly-<sha> tag
|
|
# dated 2026-09-09 or earlier — the nightly build runs ~06:16 UTC.
|
|
FN_IMAGE=vllm/vllm-openai:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657
|
|
API_KEY=replace-me
|
|
|
|
# ── Placement ───────────────────────────────────────────────────────────────
|
|
# GPU 2 was completely idle (2 MiB) before this seat; GPU 3 still is. Every other
|
|
# compose GPU pin on fv-ml1 is 0 or 1, so this seat displaced nothing.
|
|
FN_GPU_ID=2
|
|
FN_PORT=8022
|
|
FN_CONTAINER_NAME=vllm-flash-next
|
|
|
|
# ── Model ───────────────────────────────────────────────────────────────────
|
|
# dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b990578ef3031eccf9f28e675a289a09ee9
|
|
FN_MODEL=/tank/aimodels/qwen38-flash-next-abliterated-nvfp4
|
|
FN_QUANT=modelopt_fp4
|
|
FN_SERVED_NAME=qwen3.8-flash-next-uncensored
|
|
FN_SERVED_NAME_THINK=qwen3.8-flash-next-uncensored-thinking
|
|
|
|
# ── The offload ─────────────────────────────────────────────────────────────
|
|
# The 51B n-gram table (47.7 GiB FP8) lives in pinned host RAM; the GPU reads rows
|
|
# over CUDA UVA. Without this the checkpoint needs ~126 GiB of VRAM and will not
|
|
# start on one 96 GiB card. fv-ml1 has 566 GB RAM / ~388 GB available, so the
|
|
# host side is not a constraint here — unlike every DGX-Spark report upstream,
|
|
# where host and device share one unified pool and "offload" frees nothing.
|
|
FN_ENGRAM_CONFIG={"cpu_offload": true}
|
|
|
|
# ── Memory and context ──────────────────────────────────────────────────────
|
|
# FIRST-BOOT. Card is dedicated, so 0.90 (the vLLM recipe's own figure for offload
|
|
# configs) is not competing with a neighbour. Expected resident weights ~78 GiB of
|
|
# 95.6, leaving ~8-9 GiB for KV + activation inside the 0.90 budget.
|
|
# ⚠ NEXT STEP after first boot: read vLLM's budget line and replace this with an
|
|
# explicit --kv-cache-memory in BYTES. A ratio sizes the KV calculation but does
|
|
# not bound peak activation — that is what bit stacks/mog-sec twice on 2026-09-10.
|
|
FN_GPU_MEM_UTIL=0.90
|
|
|
|
# FIRST-BOOT. 128K, against a 262,144 native ceiling. Raising this is gated on a
|
|
# depth bisect with a NON-REPEATING prompt, because two open upstream issues make
|
|
# depth the risky axis: #54764 (PLE short-conv prefill pads every request in a
|
|
# batch to the batch-max query length) and #54919 (long prefill starving decode
|
|
# for 3-7 minutes). See the compose comment.
|
|
FN_MAX_MODEL_LEN=131072
|
|
FN_MAX_NUM_SEQS=16
|
|
FN_MAX_NUM_BATCHED_TOKENS=8192
|
|
FN_MAMBA_CACHE_MODE=align
|
|
|
|
# ── Prefix caching ──────────────────────────────────────────────────────────
|
|
# ⚠ THE ROLLBACK LEVER for open #54173 (CUBLAS_STATUS_INTERNAL_ERROR / illegal
|
|
# memory access in the GDN path, WITH prefix caching). Set to the empty string to
|
|
# disable. Leave FN_MAMBA_CACHE_MODE=align either way — Qwen4Exp raises on "all".
|
|
FN_PREFIX_CACHING=--enable-prefix-caching
|
|
|
|
# ── Vision ──────────────────────────────────────────────────────────────────
|
|
# 4194304 px = 2048x2048 -> ~5,125 image tokens. The checkpoint's own preprocessor
|
|
# declares 16777216 (4096x4096) -> ~16,384 tokens, which is both wasteful and fatal
|
|
# on builds enforcing the image-token count check. Same trap as stacks/mog-sec.
|
|
FN_MM_PROCESSOR_KWARGS={"size": {"longest_edge": 4194304, "shortest_edge": 65536}}
|
|
FN_LIMIT_MM={"image": 4}
|
|
|
|
# ── Misc ────────────────────────────────────────────────────────────────────
|
|
FN_REASONING_PARSER=qwen3
|
|
FN_REASONING_EFFORT=medium
|
|
FN_TOOL_CALL_PARSER=qwen3_xml
|
|
# Intentionally EMPTY. expandable_segments has corrupted retained tensors on this
|
|
# box before (quant playbook §3.10) and has never been tested against a pinned
|
|
# host allocation handed to UVA.
|
|
FN_ALLOC_CONF=
|
|
|
|
# ── NOT SET, on purpose ─────────────────────────────────────────────────────
|
|
# --speculative-config : MTP is off. vLLM's own recipe measured it WORSE at every
|
|
# concurrency on 4xH100 (8-36% less throughput, 32-173%
|
|
# more latency, ~36% acceptance) and open #55357 reports
|
|
# episodic 0% acceptance with repetition collapse.
|
|
# --kv-cache-dtype fp8 : fp8_e4m3 KV on this model's QSA path is an unmerged RFC
|
|
# (#54426). Do not copy it over from gen/mog-sec.
|