vh
4390be947d
Swaps gen-large from the dealignai ModelOpt W4A4 build to orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4, which is weight-only on both axes (W8 float attn, W4 float experts, input_activations: null) and so avoids the 4-bit-activation long-context degradation mode. The checkpoint was previously recorded as unloadable on any mainline vLLM, requiring a from-source PLE-loader patch. That conclusion was wrong on cost. Qwen4ExpPLEEmbeddingMethod.from_quant_config checks ple_embedding_dtype as branch 1, before any quant-config type check, and its NotImplementedError for CompressedTensorsConfig is scoped to the PLE path only -- experts and dense load through the ordinary compressed-tensors paths. Verified by instantiating the real config and calling the selector both ways before doing any work. orcarouter ships a bf16 PLE, so the fix was to make the declaration true: convert the 51.2B-param table to FP8 and declare it. Its 128 PLE tensors sit in one shard file with nothing else in it. Global amax 0.0894, per-shard outlier ratio 1.66x, scale chosen exactly representable in bf16 so no scale-rounding error stacks on quantization; amax maps to 446.17/448, no clipping. Round-trip 2.655% RMS relative, 0.002% underflow, 0 saturation -- the same FP8-PLE treatment dealignai already shipped. MTP head (31 tensors) and vision tower carried through untouched. A second, independent blocker followed: orcarouter labels its 12 QSA layers qwen_sparse_attention, which vLLM rejects; it accepts full_attention and selects QSA via indexer_n_heads. Confirmed indexer_n_heads == 4 in both this and the dealignai checkpoint before renaming -- without that check the rename silently selects plain attention and serves a subtly wrong model that still passes a healthcheck. Measured on the live seat: healthy, coherent, KV 344,155 tokens @ 262,144 ctx, MTP k=3 at 60.4% acceptance / 2.81 mean acceptance length, warm decode median 167.5 tok/s at conc=1 (n=5, spread 12.2%). The reorg note's dealignai figure came from a different harness, so this is not claimed as a win over it; what it does establish is that weight-only experts did not cost decode speed. Still open: controlled quality A/B vs dealignai, and a deep-prefill probe at 262K. Rollback is two .env keys; dealignai remains on disk. Also corrects the README's MTP-is-off section, stale since k=3 was deployed, and adds a superseded-claims row to the quantization playbook.
112 lines
7.6 KiB
Bash
112 lines
7.6 KiB
Bash
# flash-next-seat — Qwen3.8-Flash-Next (abliterated) on fv-ml1 GPU 2, :8022.
|
|
# Copy to .env on the host at /opt/docker/compose/flash-next-seat/.env.
|
|
#
|
|
# This is the INITIAL configuration, stood up 2026-09-13. Values marked FIRST-BOOT
|
|
# are deliberately conservative and expected to be revised once the seat has
|
|
# reported its own memory budget and been bisected for depth. Do not treat them as
|
|
# measured — they are not yet.
|
|
|
|
# ── Image ───────────────────────────────────────────────────────────────────
|
|
# ⚠ MUST contain vLLM #54371 (UVA PLE-offload), merged 2026-09-09T14:32Z.
|
|
# Verified by ancestry rather than version string: this commit is +150 / behind_by=0
|
|
# from merge commit 3116c5d06bfe76501b3dd6b5434bfc7f3274f5e7. v0.29.0 does NOT
|
|
# contain it (cut ~6h before the merge) and neither does any nightly-<sha> tag
|
|
# dated 2026-09-09 or earlier — the nightly build runs ~06:16 UTC.
|
|
FN_IMAGE=vllm/vllm-openai:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657
|
|
API_KEY=replace-me
|
|
|
|
# ── Placement ───────────────────────────────────────────────────────────────
|
|
# GPU 2 was completely idle (2 MiB) before this seat; GPU 3 still is. Every other
|
|
# compose GPU pin on fv-ml1 is 0 or 1, so this seat displaced nothing.
|
|
FN_GPU_ID=2
|
|
FN_PORT=8022
|
|
FN_CONTAINER_NAME=vllm-flash-next
|
|
|
|
# ── Model ───────────────────────────────────────────────────────────────────
|
|
# orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4, with its bf16 PLE converted to FP8
|
|
# in-house (2026-09-14) so vLLM's from_quant_config branch 1 selects the FP8 PLE method.
|
|
# ⚠ compressed-tensors, NOT modelopt_fp4 — this checkpoint is weight-only on BOTH axes
|
|
# (W8 float attn / W4 float experts, input_activations: null), which is why it was chosen
|
|
# over the dealignai W4A4 build. Rollback: the dealignai pair below, both still on disk.
|
|
# FN_MODEL=/tank/aimodels/qwen38-flash-next-abliterated-nvfp4 FN_QUANT=modelopt_fp4
|
|
FN_MODEL=/tank/aimodels/qwen38-flash-next-orcarouter-nvfp4-plefp8
|
|
FN_QUANT=compressed-tensors
|
|
FN_SERVED_NAME=qwen3.8-flash-next-uncensored
|
|
FN_SERVED_NAME_THINK=qwen3.8-flash-next-uncensored-thinking
|
|
|
|
# ── The offload ─────────────────────────────────────────────────────────────
|
|
# The 51B n-gram table (47.7 GiB FP8) lives in pinned host RAM; the GPU reads rows
|
|
# over CUDA UVA. Without this the checkpoint needs ~126 GiB of VRAM and will not
|
|
# start on one 96 GiB card. fv-ml1 has 566 GB RAM / ~388 GB available, so the
|
|
# host side is not a constraint here — unlike every DGX-Spark report upstream,
|
|
# where host and device share one unified pool and "offload" frees nothing.
|
|
FN_ENGRAM_CONFIG={"cpu_offload": true}
|
|
|
|
# Card is dedicated -- one tenant, nothing to compete with. Measured at 262K:
|
|
# weights 74.36 GiB resident, 560,654 KV tokens, 2.14x concurrency.
|
|
# ⚠ This ratio is now ADVISORY ONLY -- see the KV pin below, which overrides it.
|
|
FN_GPU_MEM_UTIL=0.96
|
|
# ⚠⚠ 14.00 GiB, PINNED IN BYTES AND MEASURED THE HARD WAY (2026-09-13).
|
|
# 16.00 GiB was tried first and nearly OOM'd: a 155,497-token prefill drove GPU 2 to
|
|
# 97,074 of 97,887 MiB and the allocator logged "OOM on device 0 while trying to
|
|
# allocate 488636416 bytes (free: 422117376)" -- 466 MiB wanted, 403 MiB free. The
|
|
# request survived but that is one step before the mog-sec crash shape.
|
|
# ⚠ WHY THE ESTIMATE WAS WRONG: --kv-cache-memory makes vLLM SKIP MEMORY PROFILING
|
|
# and ignore --gpu-memory-utilization entirely. The profiler was the thing accounting
|
|
# for deep-prefill activation; pinning bytes turns it off. Do NOT take vLLM's
|
|
# "17.46 GiB to fully utilize" suggestion -- it is computed from a profile taken at
|
|
# max-num-batched-tokens depth and is 3.5 GiB above what a 150K request survives.
|
|
# Re-raising requires re-running the deep probe and reading the allocator log.
|
|
# VERIFIED at 14.00 GiB: 0 OOM warnings, 155,557-token prefill in 14.2 s, 2,085 MiB
|
|
# still free on the card at peak.
|
|
# ⚠ MTP ON (k=3) REQUIRES A SMALLER KV POOL THAN THE NO-SPEC SEAT.
|
|
# Measured 2026-09-13: the draft head adds ~5.08 GiB of weights and raises
|
|
# per-token KV cost ~16%. At the old 15032385536 (14 GiB) the engine OOMs at
|
|
# init with MTP enabled -- 79.44 GiB weights + 14 GiB KV = 93.44 against a
|
|
# 91.17 GiB budget. 8589934592 (8 GiB) is PROVEN to boot; 10737418240 (10 GiB)
|
|
# leaves ~1.73 GiB for activations and is the current setting. If the seat
|
|
# OOMs at init, drop to 8589934592.
|
|
FN_KV_CACHE_MEMORY=10737418240
|
|
|
|
# FULL NATIVE 262,144 (operator-directed 2026-09-13). The KV pool holds ~641K
|
|
# tokens, so a single max-length request fits with ~2.4x concurrency to spare.
|
|
# ⚠ STARTUP IS NOT A DEPTH TEST. Two open upstream issues make depth the risky
|
|
# axis -- #54764 (PLE short-conv prefill pads every request in a batch to the
|
|
# batch-MAX query length) and #54919 (long prefill starving active decode for 3-7
|
|
# minutes) -- and vLLM's own recipe admits a single 262K request was never tested.
|
|
# If deep requests misbehave, --max-num-batched-tokens is the lever, not this.
|
|
FN_MAX_MODEL_LEN=262144
|
|
FN_MAX_NUM_SEQS=16
|
|
FN_MAX_NUM_BATCHED_TOKENS=8192
|
|
FN_MAMBA_CACHE_MODE=align
|
|
|
|
# ── Prefix caching ──────────────────────────────────────────────────────────
|
|
# ⚠ THE ROLLBACK LEVER for open #54173 (CUBLAS_STATUS_INTERNAL_ERROR / illegal
|
|
# memory access in the GDN path, WITH prefix caching). Set to the empty string to
|
|
# disable. Leave FN_MAMBA_CACHE_MODE=align either way — Qwen4Exp raises on "all".
|
|
FN_PREFIX_CACHING=--enable-prefix-caching
|
|
|
|
# ── Vision ──────────────────────────────────────────────────────────────────
|
|
# 4194304 px = 2048x2048 -> ~5,125 image tokens. The checkpoint's own preprocessor
|
|
# declares 16777216 (4096x4096) -> ~16,384 tokens, which is both wasteful and fatal
|
|
# on builds enforcing the image-token count check. Same trap as stacks/mog-sec.
|
|
FN_MM_PROCESSOR_KWARGS={"size": {"longest_edge": 4194304, "shortest_edge": 65536}}
|
|
FN_LIMIT_MM={"image": 4}
|
|
|
|
# ── Misc ────────────────────────────────────────────────────────────────────
|
|
FN_REASONING_PARSER=qwen3
|
|
FN_REASONING_EFFORT=medium
|
|
FN_TOOL_CALL_PARSER=qwen3_xml
|
|
# Intentionally EMPTY. expandable_segments has corrupted retained tensors on this
|
|
# box before (quant playbook §3.10) and has never been tested against a pinned
|
|
# host allocation handed to UVA.
|
|
FN_ALLOC_CONF=
|
|
|
|
# ── NOT SET, on purpose ─────────────────────────────────────────────────────
|
|
# --speculative-config : MTP is off. vLLM's own recipe measured it WORSE at every
|
|
# concurrency on 4xH100 (8-36% less throughput, 32-173%
|
|
# more latency, ~36% acceptance) and open #55357 reports
|
|
# episodic 0% acceptance with repetition collapse.
|
|
# --kv-cache-dtype fp8 : fp8_e4m3 KV on this model's QSA path is an unmerged RFC
|
|
# (#54426). Do not copy it over from gen/mog-sec.
|