vh
1b5d6ba23a
Operator instruction: delete the displaced dealignai checkpoint. 125 GiB reclaimed from /tank (59% -> 57% used). Verified before removing: not mounted by any running or exited container, no symlinks, no inodes shared with the converted orcarouter directory. Every "rollback is two .env keys" statement across the stack README, the .env.example, persistent-memory and its detail file was true when written and is false now -- the .env backup still names paths that no longer exist. Corrected in place rather than left as false reassurance, since a stale rollback instruction is discovered precisely when it is needed. Reverting this seat now costs a 126 GiB re-download. The quality A/B against dealignai is likewise no longer runnable locally: its reference arm is gone. The pristine 170 GiB orcarouter download is retained deliberately -- it is what makes the PLE bf16->FP8 conversion reproducible without re-fetching -- and that is now recorded so a future session does not reclaim it as an obvious duplicate. Also notes that ~75 GiB of non-PLE shards are duplicated between the pristine and converted orca directories (the convert's hardlinks hit EXDEV across two container bind mounts); both now sit directly on /tank, so relinking would reclaim it if /tank ever tightens.
187 lines
11 KiB
Markdown
187 lines
11 KiB
Markdown
# flash-next-seat — Qwen3.8-Flash-Next (abliterated), fv-ml1 GPU 2, `:8022`
|
||
|
||
The first seat on the fleet whose weights do not fit its card and run anyway.
|
||
|
||
`Qwen3.8-Flash-Next` is 176B total — a 125B main model plus a **51B n-gram (PLE)
|
||
lookup table** — activating ~6B parameters per token. The n-gram table is a pure
|
||
embedding lookup with almost no compute per token, so it lives in **pinned host
|
||
RAM** and the GPU reads the rows it needs directly over **CUDA UVA** on a dedicated
|
||
stream with async prefetch.
|
||
|
||
| | |
|
||
|---|---|
|
||
| Checkpoint | `orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4`, PLE converted bf16→FP8 in-house 2026-09-14 (123.2 GiB) |
|
||
| On the card | ~75 GiB of 95.6 GiB — **weight-only on both axes**: W4 float experts, W8 float attn, `input_activations: null` |
|
||
| In host RAM | 47.7 GiB pinned, FP8 E4M3, 8 `model-plefp8-*` shards + one global BF16 scale |
|
||
| Context | **262,144** (full native) — KV 344,155 tokens, 1.31x concurrency |
|
||
| Speculative decoding | **MTP k=3** — 60.4% acceptance, mean acceptance length 2.81 (measured here, n=5) |
|
||
| Gateway wiring | **8 aliases** — gen, gen-reasoning, summarizer(-large), classifier, chat-judge, image-judge, qwen-image-bench |
|
||
|
||
## Deploy
|
||
|
||
```bash
|
||
scripts/deploy-stack.sh fv-ml1 flash-next-seat # diffs vs live, prompts y/N
|
||
# then on the host, first boot only:
|
||
ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/flash-next-seat && docker compose config >/dev/null && docker compose up -d'
|
||
```
|
||
|
||
The `.env` lives on the host and is never committed. Copy `.env.example`, set
|
||
`API_KEY`, and read the FIRST-BOOT annotations before changing anything else.
|
||
|
||
## Architecture, briefly
|
||
|
||
Four ideas, and three of them shape the serving config:
|
||
|
||
- **GDN + QSA.** 36 of 48 layers use Gated DeltaNet (linear attention) to compress
|
||
history; every fourth layer uses Qwen Sparse Attention for long-range retrieval.
|
||
This is why KV is cheap at depth and why `--mamba-cache-dtype float32` matters.
|
||
- **N-gram embedding.** The 51B lookup table that this seat offloads. Qwen's own
|
||
framing: capacity with almost no per-token compute.
|
||
- **Gated residual / hyper-connections.** Four residual branches; excluded from
|
||
quantization in this checkpoint.
|
||
- **MTP head.** Present, preserved byte-identically, and **in use at k=3**.
|
||
|
||
## Why this checkpoint, and the two traps in front of it
|
||
|
||
Chosen for the **activation axis**: orcarouter's build is weight-only on *both* halves —
|
||
`config_groups` gives W8 float for attention/dense and W4 float for the experts, with
|
||
`input_activations: null` on each. The displaced dealignai build is ModelOpt **W4A4**
|
||
(4-bit activations), the long-context degradation mode. Same author as the `gen` seat.
|
||
|
||
It did not load out of the box, and there were **two independent config-level blockers**.
|
||
Both are recorded here because each looks like a capability gap and neither is one.
|
||
|
||
### Trap 1 — the PLE loader (and the claim we had wrong)
|
||
|
||
vLLM picks the PLE table's format in `Qwen4ExpPLEEmbeddingMethod.from_quant_config`:
|
||
|
||
```
|
||
1. ple_embedding_dtype == "float8_e4m3fn" -> FP8 method <-- BEFORE any type check
|
||
2. quant_config is None -> unquantized
|
||
3. ModelOptMixedPrecisionConfig -> FP8 / unquantized
|
||
4. ModelOptQuantConfigBase + excluded -> unquantized
|
||
5. not isinstance(quant_config, Fp8Config)-> NotImplementedError
|
||
```
|
||
|
||
⚠ **This README previously said an FP8 PLE without the declaration is disqualifying, and
|
||
that compressed-tensors needs a vLLM source patch. Both were wrong** (corrected 2026-09-14;
|
||
see the quantization playbook's superseded-claims table). Branch 1 is **unconditional**, and
|
||
the `NotImplementedError` is **scoped to the PLE path only** — experts and dense layers of a
|
||
compressed-tensors build load through vLLM's ordinary compressed-tensors paths. So declaring
|
||
an FP8 PLE bypasses the blocker on stock mainline.
|
||
|
||
orcarouter ships a **bf16** PLE, so the honest fix was to *make the declaration true*:
|
||
convert the table to FP8, then declare it. Its 128 PLE tensors sit in exactly one shard file
|
||
with nothing else in it, which makes that a clean, cheap rewrite.
|
||
|
||
⚠ **Declare only what is true.** `gorbatjovy/...-NVFP4-plefp8` ships an FP8 table with no
|
||
declaration and dies on `ngram_embedding.weight_scale`; declaring FP8 over a *bf16* table is
|
||
that same failure in reverse. The declaration is a claim about the bytes, not a switch.
|
||
|
||
### Trap 2 — `Invalid layer_type qwen_sparse_attention`
|
||
|
||
orcarouter labels its 12 QSA layers `qwen_sparse_attention`. vLLM accepts only
|
||
`linear_attention` and `full_attention`, and selects QSA *within* `full_attention` when
|
||
`indexer_n_heads` is present. The fix is renaming the 12 entries.
|
||
|
||
⚠⚠ **Check `indexer_n_heads` before renaming.** Without it the rename silently selects plain
|
||
`Qwen3NextAttention` instead of `Qwen4ExpQSAAttention` — a subtly wrong model that loads,
|
||
serves, and passes a healthcheck. Verified `indexer_n_heads == 4` in both this checkpoint and
|
||
the dealignai one, along with every other indexer/QSA key, before touching it.
|
||
|
||
### The conversion, and what it cost
|
||
|
||
Global amax 0.0894 with a per-shard outlier ratio of only **1.66x**, so the single global
|
||
scale this method uses is well-conditioned here. The scale is chosen **exactly representable
|
||
in bf16** (2.002716e-04) so no scale-rounding error stacks on the quantization error; amax
|
||
maps to 446.17 of 448, so nothing clips. Round-trip **2.655% RMS relative**, 0.002% underflow,
|
||
zero saturation — and the same FP8-PLE treatment dealignai already shipped, so it is not a
|
||
regression against the seat it replaced. `weight_scale` is written BF16 [1] to match the
|
||
published format. MTP head (31 tensors) and the vision tower carry through untouched.
|
||
|
||
⚠⚠ **THERE IS NO LOCAL ROLLBACK.** The dealignai checkpoint was deleted on operator
|
||
instruction 2026-09-14 (125 GiB reclaimed). Reverting this seat now means **re-downloading
|
||
126 GiB** from `dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4`, not flipping two `.env`
|
||
keys. The `.env` backup (`.env.bak-preorca-20260914-023408`) still names the old paths, but
|
||
those paths no longer exist — treat it as a record of the old settings, not a working revert.
|
||
|
||
⚠ Still unmeasured, and now unbacked: a controlled quality A/B against dealignai — which was
|
||
the entire reason for the swap, and whose reference arm is gone — and a deep-prefill probe at
|
||
262K on this checkpoint. The 170 GiB pristine `qwen38-flash-next-orcarouter-nvfp4` download is
|
||
retained; it is what lets the PLE conversion be redone without re-fetching.
|
||
|
||
Rejected alternatives, for the record: `nvidia/…-NVFP4` is the cleanest ModelOpt build but is
|
||
not abliterated; `lovedheart/…-Pruned-RTXPRO-6000` prunes to 448 of 512 experts;
|
||
`windowsxp811203/…-Abliterated-NVFP4` stores its 95 GiB PLE as a single malformed
|
||
`ple_embedding.shard_.weight` instead of 128 `ngram_embedding.shard_N.weight` and has never
|
||
been served by its own author.
|
||
|
||
## Why MTP is ON at k=3 (reversing this seat's original default)
|
||
|
||
This seat shipped with speculative decoding off, citing vLLM's recipe: on 4xH100 at TP=4
|
||
that recipe measured MTP **worse at every concurrency** (8-36% lower throughput, 32-173%
|
||
higher per-token latency, ~36% acceptance) and says do not default it on. Open #55357
|
||
reports episodic 0% acceptance with repetition collapse inside thinking blocks.
|
||
|
||
**Measured here, that inverted.** The campaign in `services/flash-next-mtp-bench/` found MTP
|
||
a win at every k and every concurrency tested on one Blackwell card (+29/41/27% at k=1,
|
||
+42/52/38% at k=2, +52/51/34% at k=3 across conc 1/4/8). k=3 is deployed because this is a
|
||
single-user fleet and conc=1 dominates.
|
||
|
||
On the current orcarouter checkpoint, measured 2026-09-14: **60.4% acceptance, mean
|
||
acceptance length 2.81** (per-position 80.6 / 60.8 / 40.8%), warm decode median **167.5
|
||
tok/s** at conc=1 (n=5, spread 12.2%).
|
||
|
||
⚠ **MTP costs KV.** The draft head adds ~5.08 GiB of weights and raises per-token KV cost
|
||
~16%; `FN_KV_CACHE_MEMORY` was cut 14 -> 10 GiB for it. At 14 GiB the engine OOMs at init
|
||
with MTP on. If it OOMs, drop to 8589934592.
|
||
|
||
⚠ The recipe's numbers are someone else's hardware, and so are ours to anyone else. Re-measure
|
||
on the seat, warm, with repeats — the first decode bench during the reorg read 39 tok/s and
|
||
that was a cold-boot + contention artifact, not a result.
|
||
|
||
## The upstream situation, as of 2026-09-13
|
||
|
||
- **#53896** — model support. **Merged 2026-08-31.** In v0.29.0.
|
||
- **#54371** — *UVA PLE-offload and Engram tensor parallelism*. **Merged
|
||
2026-09-09T14:32Z.** This is the offload this seat uses. **Not in v0.29.0**,
|
||
which was cut ~6 h earlier; present in `v0.29.1rc0` and in any nightly from
|
||
2026-09-10 onward.
|
||
- **#53899** — the *older, worker-based* PLE offload. **Open and explicitly paused**
|
||
in favour of #54371. Do not go back to it. Its whole bug family — the TP=1
|
||
startup deadlock (#53960), the `pidfd_getfd` / `kernel.yama.ptrace_scope` gate,
|
||
the shared-CUDA-event race under async scheduling, and silently one-step-stale
|
||
PLE outputs under CUDA graphs — came from the separate worker process and the
|
||
CUDA-IPC row transfer that the UVA path does not have.
|
||
|
||
Open issues worth knowing about on SM120, none of them blocking:
|
||
|
||
| Issue | What it does | Our exposure |
|
||
|---|---|---|
|
||
| **#54173** | CUBLAS internal error / illegal memory access in the GDN path **with prefix caching** | We enable prefix caching. `FN_PREFIX_CACHING=` is the one-line rollback. |
|
||
| **#54764** | PLE short-conv batched prefill pads every request to the batch-max query length | Why `--max-num-batched-tokens` is 8192, not 16384 |
|
||
| **#54919** | Long prefill starves active decode for 3–7 minutes | Why context starts at 128K |
|
||
| **#54521** | Greedy decoding non-deterministic from `persistent_topk` in prefill | Affects any A/B on this seat — establish a noise floor before comparing |
|
||
| **#54426** | fp8_e4m3 KV on the QSA path is an unmerged RFC | Why `--kv-cache-dtype` is **not** set to fp8 here |
|
||
|
||
## Raising context
|
||
|
||
128K is a starting value, not a measured one. Before raising it, bisect with a
|
||
**non-repeating** prompt — a repeated one hashes to cached blocks and never
|
||
prefills deep, so it proves nothing. The `stacks/mog-sec` README records this the
|
||
hard way: three successive context cuts all sized the *KV pool* while the crashes
|
||
were governed by *processing depth*, which is a different number.
|
||
|
||
The point of a ceiling is the refusal. Below it the seat serves; above it vLLM
|
||
returns a clean 400 naming the limit, instead of the engine dying and taking every
|
||
in-flight request with it.
|
||
|
||
## Not done yet
|
||
|
||
- **Pin `--kv-cache-memory` in bytes** from the first boot's budget line, replacing
|
||
the 0.90 ratio. Same discipline as `stacks/mog-sec` and `stacks/erp-seat`.
|
||
- **Gateway wiring is deliberately absent.** Pointing any LiteLLM alias at this
|
||
seat — in particular displacing `gen` / `summarizer` / `classifier`, which is the
|
||
long-term intent recorded in henge item 49 — changes what every existing caller
|
||
receives and is the operator's call, not a deploy-time default.
|