217 lines
13 KiB
Markdown
217 lines
13 KiB
Markdown
# flash-next-seat — Qwen3.8-Flash-Next (abliterated), fv-ml1 GPU 2, `:8022`
|
||
|
||
The first seat on the fleet whose weights do not fit its card and run anyway.
|
||
|
||
`Qwen3.8-Flash-Next` is 176B total — a 125B main model plus a **51B n-gram (PLE)
|
||
lookup table** — activating ~6B parameters per token. The n-gram table is a pure
|
||
embedding lookup with almost no compute per token, so it lives in **pinned host
|
||
RAM** and the GPU reads the rows it needs directly over **CUDA UVA** on a dedicated
|
||
stream with async prefetch.
|
||
|
||
| | |
|
||
|---|---|
|
||
| Checkpoint | `orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4`, PLE converted bf16→FP8 in-house 2026-09-14 (123.2 GiB) |
|
||
| On the card | ~75 GiB of 95.6 GiB — **weight-only on both axes**: W4 float experts, W8 float attn, `input_activations: null` |
|
||
| In host RAM | 47.7 GiB pinned, FP8 E4M3, 8 `model-plefp8-*` shards + one global BF16 scale |
|
||
| Context | **262,144** (full native) — KV 344,155 tokens, 1.31x concurrency |
|
||
| Speculative decoding | **MTP k=3** — 60.4% acceptance, mean acceptance length 2.81 (measured here, n=5) |
|
||
| Gateway wiring | **8 aliases** — gen, gen-reasoning, summarizer(-large), classifier, chat-judge, image-judge, qwen-image-bench |
|
||
|
||
> **Upstream check, 2026-09-26:** orcarouter's only commit since our 2026-09-14
|
||
> download (`c1209bda`) is `38efbff` (2026-09-18). It **adds `repack_sm70.py`**, a
|
||
> helper that repacks the model for V100 (experts-only NVFP4, bf16 elsewhere), and
|
||
> nothing else. Every weight, config and card file has the same content hash, so
|
||
> there is nothing to adopt. Upstream still ships a bf16 PLE and
|
||
> `qwen_sparse_attention` layer names, so both in-house fixes below remain
|
||
> required. Re-check with the HF tree API (`/api/models/<repo>/tree/<rev>`,
|
||
> gated, needs `$HF_TOKEN`) rather than re-downloading.
|
||
|
||
## Deploy
|
||
|
||
```bash
|
||
scripts/deploy-stack.sh fv-ml1 flash-next-seat # diffs vs live, prompts y/N
|
||
# then on the host, first boot only:
|
||
ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/flash-next-seat && docker compose config >/dev/null && docker compose up -d'
|
||
```
|
||
|
||
The `.env` lives on the host and is never committed. Copy `.env.example`, set
|
||
`API_KEY`, and read the FIRST-BOOT annotations before changing anything else.
|
||
|
||
## Architecture, briefly
|
||
|
||
Four ideas, and three of them shape the serving config:
|
||
|
||
- **GDN + QSA.** 36 of 48 layers use Gated DeltaNet (linear attention) to compress
|
||
history; every fourth layer uses Qwen Sparse Attention for long-range retrieval.
|
||
This is why KV is cheap at depth and why `--mamba-cache-dtype float32` matters.
|
||
- **N-gram embedding.** The 51B lookup table that this seat offloads. Qwen's own
|
||
framing: capacity with almost no per-token compute.
|
||
- **Gated residual / hyper-connections.** Four residual branches; excluded from
|
||
quantization in this checkpoint.
|
||
- **MTP head.** Present, preserved byte-identically, and **in use at k=3**.
|
||
|
||
## Why this checkpoint, and the two traps in front of it
|
||
|
||
Chosen for the **activation axis**: orcarouter's build is weight-only on *both* halves —
|
||
`config_groups` gives W8 float for attention/dense and W4 float for the experts, with
|
||
`input_activations: null` on each. The displaced dealignai build is ModelOpt **W4A4**
|
||
(4-bit activations), the long-context degradation mode. Same author as the `gen` seat.
|
||
|
||
It did not load out of the box, and there were **two independent config-level blockers**.
|
||
Both are recorded here because each looks like a capability gap and neither is one.
|
||
|
||
### Trap 1 — the PLE loader (and the claim we had wrong)
|
||
|
||
vLLM picks the PLE table's format in `Qwen4ExpPLEEmbeddingMethod.from_quant_config`:
|
||
|
||
```
|
||
1. ple_embedding_dtype == "float8_e4m3fn" -> FP8 method <-- BEFORE any type check
|
||
2. quant_config is None -> unquantized
|
||
3. ModelOptMixedPrecisionConfig -> FP8 / unquantized
|
||
4. ModelOptQuantConfigBase + excluded -> unquantized
|
||
5. not isinstance(quant_config, Fp8Config)-> NotImplementedError
|
||
```
|
||
|
||
⚠ **This README previously said an FP8 PLE without the declaration is disqualifying, and
|
||
that compressed-tensors needs a vLLM source patch. Both were wrong** (corrected 2026-09-14;
|
||
see the quantization playbook's superseded-claims table). Branch 1 is **unconditional**, and
|
||
the `NotImplementedError` is **scoped to the PLE path only** — experts and dense layers of a
|
||
compressed-tensors build load through vLLM's ordinary compressed-tensors paths. So declaring
|
||
an FP8 PLE bypasses the blocker on stock mainline.
|
||
|
||
orcarouter ships a **bf16** PLE, so the honest fix was to *make the declaration true*:
|
||
convert the table to FP8, then declare it. Its 128 PLE tensors sit in exactly one shard file
|
||
with nothing else in it, which makes that a clean, cheap rewrite.
|
||
|
||
⚠ **Declare only what is true.** `gorbatjovy/...-NVFP4-plefp8` ships an FP8 table with no
|
||
declaration and dies on `ngram_embedding.weight_scale`; declaring FP8 over a *bf16* table is
|
||
that same failure in reverse. The declaration is a claim about the bytes, not a switch.
|
||
|
||
### Trap 2 — `Invalid layer_type qwen_sparse_attention`
|
||
|
||
orcarouter labels its 12 QSA layers `qwen_sparse_attention`. vLLM accepts only
|
||
`linear_attention` and `full_attention`, and selects QSA *within* `full_attention` when
|
||
`indexer_n_heads` is present. The fix is renaming the 12 entries.
|
||
|
||
⚠⚠ **Check `indexer_n_heads` before renaming.** Without it the rename silently selects plain
|
||
`Qwen3NextAttention` instead of `Qwen4ExpQSAAttention` — a subtly wrong model that loads,
|
||
serves, and passes a healthcheck. Verified `indexer_n_heads == 4` in both this checkpoint and
|
||
the dealignai one, along with every other indexer/QSA key, before touching it.
|
||
|
||
### The conversion, and what it cost
|
||
|
||
Global amax 0.0894 with a per-shard outlier ratio of only **1.66x**, so the single global
|
||
scale this method uses is well-conditioned here. The scale is chosen **exactly representable
|
||
in bf16** (2.002716e-04) so no scale-rounding error stacks on the quantization error; amax
|
||
maps to 446.17 of 448, so nothing clips. Round-trip **2.655% RMS relative**, 0.002% underflow,
|
||
zero saturation — and the same FP8-PLE treatment dealignai already shipped, so it is not a
|
||
regression against the seat it replaced. `weight_scale` is written BF16 [1] to match the
|
||
published format. MTP head (31 tensors) and the vision tower carry through untouched.
|
||
|
||
⚠⚠ **THERE IS NO LOCAL ROLLBACK.** The dealignai checkpoint was deleted on operator
|
||
instruction 2026-09-14 (125 GiB reclaimed). Reverting this seat now means **re-downloading
|
||
126 GiB** from `dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4`, not flipping two `.env`
|
||
keys. The `.env` backup (`.env.bak-preorca-20260914-023408`) still names the old paths, but
|
||
those paths no longer exist — treat it as a record of the old settings, not a working revert.
|
||
|
||
⚠ Still unmeasured, and now unbacked: a controlled quality A/B against dealignai — which was
|
||
the entire reason for the swap, and whose reference arm is gone — and a deep-prefill probe at
|
||
262K on this checkpoint. The 170 GiB pristine `qwen38-flash-next-orcarouter-nvfp4` download is
|
||
retained; it is what lets the PLE conversion be redone without re-fetching.
|
||
|
||
Rejected alternatives, for the record: `nvidia/…-NVFP4` is the cleanest ModelOpt build but is
|
||
not abliterated; `lovedheart/…-Pruned-RTXPRO-6000` prunes to 448 of 512 experts;
|
||
`windowsxp811203/…-Abliterated-NVFP4` stores its 95 GiB PLE as a single malformed
|
||
`ple_embedding.shard_.weight` instead of 128 `ngram_embedding.shard_N.weight` and has never
|
||
been served by its own author.
|
||
|
||
## Why MTP is ON at k=3 (reversing this seat's original default)
|
||
|
||
This seat shipped with speculative decoding off, citing vLLM's recipe: on 4xH100 at TP=4
|
||
that recipe measured MTP **worse at every concurrency** (8-36% lower throughput, 32-173%
|
||
higher per-token latency, ~36% acceptance) and says do not default it on. Open #55357
|
||
reports episodic 0% acceptance with repetition collapse inside thinking blocks.
|
||
|
||
**Measured here, that inverted.** The campaign in `services/flash-next-mtp-bench/` found MTP
|
||
a win at every k and every concurrency tested on one Blackwell card (+29/41/27% at k=1,
|
||
+42/52/38% at k=2, +52/51/34% at k=3 across conc 1/4/8). k=3 is deployed because this is a
|
||
single-user fleet and conc=1 dominates.
|
||
|
||
On the current orcarouter checkpoint, measured 2026-09-14: **60.4% acceptance, mean
|
||
acceptance length 2.81** (per-position 80.6 / 60.8 / 40.8%), warm decode median **167.5
|
||
tok/s** at conc=1 (n=5, spread 12.2%).
|
||
|
||
⚠ **MTP costs KV.** The draft head adds ~5.08 GiB of weights and raises per-token KV cost
|
||
~16%; `FN_KV_CACHE_MEMORY` was cut 14 -> 10 GiB for it. At 14 GiB the engine OOMs at init
|
||
with MTP on. If it OOMs, drop to 8589934592.
|
||
|
||
⚠ The recipe's numbers are someone else's hardware, and so are ours to anyone else. Re-measure
|
||
on the seat, warm, with repeats — the first decode bench during the reorg read 39 tok/s and
|
||
that was a cold-boot + contention artifact, not a result.
|
||
|
||
## The upstream situation, as of 2026-09-13
|
||
|
||
- **#53896** — model support. **Merged 2026-08-31.** In v0.29.0.
|
||
- **#54371** — *UVA PLE-offload and Engram tensor parallelism*. **Merged
|
||
2026-09-09T14:32Z.** This is the offload this seat uses. **Not in v0.29.0**,
|
||
which was cut ~6 h earlier; present in `v0.29.1rc0` and in any nightly from
|
||
2026-09-10 onward.
|
||
- **#53899** — the *older, worker-based* PLE offload. **Open and explicitly paused**
|
||
in favour of #54371. Do not go back to it. Its whole bug family — the TP=1
|
||
startup deadlock (#53960), the `pidfd_getfd` / `kernel.yama.ptrace_scope` gate,
|
||
the shared-CUDA-event race under async scheduling, and silently one-step-stale
|
||
PLE outputs under CUDA graphs — came from the separate worker process and the
|
||
CUDA-IPC row transfer that the UVA path does not have.
|
||
|
||
Open issues worth knowing about on SM120, none of them blocking:
|
||
|
||
| Issue | What it does | Our exposure |
|
||
|---|---|---|
|
||
| **#54173** | CUBLAS internal error / illegal memory access in the GDN path **with prefix caching** | We enable prefix caching. `FN_PREFIX_CACHING=` is the one-line rollback. |
|
||
| **#54764** | PLE short-conv batched prefill pads every request to the batch-max query length | Why `--max-num-batched-tokens` is 8192, not 16384 |
|
||
| **#54919** | Long prefill starves active decode for 3–7 minutes | Why context starts at 128K |
|
||
| **#54521** | Greedy decoding non-deterministic from `persistent_topk` in prefill | Affects any A/B on this seat — establish a noise floor before comparing |
|
||
| **#54426** | fp8_e4m3 KV on the QSA path is an unmerged RFC | Why `--kv-cache-dtype` is **not** set to fp8 here |
|
||
|
||
## Context depth — PROBED 2026-09-14, clean to 258,517 tokens
|
||
|
||
262,144 is the configured ceiling and it has now been bisected with a **non-repeating**
|
||
prompt (unique random hex per probe, so prefix caching cannot short-circuit the prefill —
|
||
a repeated prompt hashes to cached blocks and never prefills deep).
|
||
|
||
| prompt tokens | 31,978 | 64,154 | 128,191 | 196,172 | 240,290 | **258,517** |
|
||
|---|---|---|---|---|---|---|
|
||
| result | ok | ok | ok | ok | ok | **ok** |
|
||
|
||
**The load-bearing evidence is the engine's own allocator log: zero OOM / CUBLAS /
|
||
illegal-memory / traceback entries across the whole run** — the same detector that caught the
|
||
dealignai near-miss (`OOM on device 0 ... 466 MiB wanted, 403 MiB free`) at 155K on the
|
||
previous checkpoint. It fired then; it is silent here. The run also happened **under real
|
||
concurrent operator load**, which makes it a stricter test than a solo probe, not a weaker one.
|
||
|
||
**Positive control passed.** A mis-sized first attempt built a ~265K-token prompt and got a
|
||
clean `400` naming the limit rather than killing the engine — so the probe could detect the
|
||
failure it was looking for. (Calibration for anyone re-running it: random hex words tokenize
|
||
at **7.9 tokens/word** on this tokenizer.)
|
||
|
||
**#54919 did not reproduce.** That issue reports long prefill starving decode for 3-7 minutes;
|
||
258K prefilled in **28.9 s** (~8,900 tok/s), scaling near-linearly from 32K.
|
||
|
||
⚠⚠ **DO NOT reuse the memory-headroom half of that probe — the gauge was blind.** It sampled
|
||
`nvidia-smi` before and after each request and reported an identical 95,460 MiB / 2,427 MiB
|
||
free on *every* row. Two reasons: `--kv-cache-memory` pins the pool and the engine log says it
|
||
**"skipped memory profiling"**, so GPU usage is constant regardless of depth; and the risk is a
|
||
*transient* activation spike **during** prefill, which before/after bracketing structurally
|
||
cannot see. Identical readings across a 8x range of depths are the tell. Real peak-activation
|
||
headroom needs in-process sampling during the prefill. The pass/fail result stands on the
|
||
allocator log, not on that column.
|
||
|
||
## Not done yet
|
||
|
||
- **Pin `--kv-cache-memory` in bytes** from the first boot's budget line, replacing
|
||
the 0.90 ratio. Same discipline as `stacks/mog-sec` and `stacks/erp-seat`.
|
||
- **Gateway wiring is deliberately absent.** Pointing any LiteLLM alias at this
|
||
seat — in particular displacing `gen` / `summarizer` / `classifier`, which is the
|
||
long-term intent recorded in henge item 49 — changes what every existing caller
|
||
receives and is the operator's call, not a deploy-time default.
|