vh
a91b841d86
The seat documentation must stay current, and a hand-written document cannot. The LiteLLM config described char-rp as a 31B model on a host and GPU it had not been on since 2026-08-24 -- three weeks of silent drift in a file that read as authoritative, and the reason a seat spent that period serving a model nobody intended. Anything typed here drifts the same way; anything read off the running containers cannot. scripts/seat-inventory.py derives the whole document from the host: - placement and VRAM from nvidia-smi compute-apps, mapped to containers through /proc/<pid>/cgroup -- nvidia-smi reports the vLLM engine child while docker reports the container pid, so matching them directly silently yields nothing - weights and KV tokens parsed from each engine's own startup log, not derived arithmetically, with concurrency computed as KV tokens over context - architecture, layer and expert counts, and the exact quantization group scheme (W4A4 vs W4A16 distinguished) from each model's config.json - speculative-decoding method and k from the container argv, which is how the three incompatible methods on this box became visible - lineage from the .PROVENANCE.txt SIBLING files -- they sit beside the model directory, not inside it, which is why an earlier pass wrongly reported two fully-documented seats as having no provenance - gateway aliases resolved from the LiteLLM config on ana-docker --check compares the committed document against the live box and exits non-zero when they diverge, ignoring only the generation timestamp. Suitable for CI or a scheduled drift alarm; read-only throughout, safe against production. Also commits the KV_CACHE_BYTES override added to the MTP campaign runner, which asserts the flag exists in the derived argv and aborts rather than running a campaign that silently ignored it.
Does MTP actually hurt Qwen3.8-Flash-Next on one Blackwell card?
Status: campaign built, running. Results and a verdict land here with the raw JSON alongside, so the conclusion can be re-derived rather than taken on faith.
Why this exists
The flash-next-seat stack shipped with MTP speculative decoding off, citing
vLLM's published recipe: on 4×H100 it measured MTP as worse at every concurrency
tested — 8–36% lower request throughput, 32–173% higher per-token latency, ~36%
acceptance — and says don't default it on.
That was not valid evidence about our seat, and defaulting on it was the wrong call. Our own measurement rule says the harness is part of the number and that cross-harness comparisons are invalid, not merely noisy. The recipe's harness differs from ours on nearly every axis that could plausibly drive the result:
| vLLM recipe | this seat | |
|---|---|---|
| GPUs | 4× H100 80 GB (Hopper) | 1× RTX PRO 6000 96 GB (Blackwell, sm_120) |
| Parallelism | TP=4 — experts sharded 4 ways, all-reduce per layer | TP=1 — every expert local, no collective |
| PLE table | offloaded (forced: 80 GB cards can't hold it) | offloaded (chosen) |
| k tested | 3 only | 1, 2, 3 |
Two mechanisms that could make MTP lose here, and one that makes k the real question
Spec decoding's usual win is that verifying k+1 tokens costs about the same as decoding 1, because decode is memory-bandwidth-bound: you read the weights once either way. Two things about this model break that assumption.
- Sparse-MoE expert-read amplification. 512 experts, 10 active per token. At batch 1, decoding one token touches ~10 experts. Verifying k+1 tokens routes each position to its own 10, largely disjoint — so the weights read scale with the token count instead of staying flat. The "free verification" premise does not hold for an ultra-sparse MoE. This is worst at low concurrency and shrinks as the batch already touches many experts anyway, which is exactly why the sweep has to cross concurrency and not just report one number.
- PLE/UVA fetch amplification. The 51B n-gram table lives in host RAM and is read over PCIe. Every token position needs its own row lookups, so k+1 tokens means k+1× the host round-trips — multiplying traffic on the single slowest link in the system, the one we deliberately moved off the card.
And the mechanism that makes k the actual experiment: the checkpoint's
mtp_num_hidden_layers is 1. The draft head is a single module run
autoregressively for k>1 — the pattern our own quant playbook §5.1 flags, where
deeper k improves acceptance and destroys throughput. If throughput falls
monotonically in k while acceptance rises, then k=1 may win and the recipe's
k=3 number says nothing about it. Nobody has published k=1 for this model.
So there are coherent reasons MTP could genuinely lose here — and an equally coherent reason the published number is the wrong number to decide on.
Design
Five boots. Arms in order: off_A, k1, k2, k3, off_B.
- Repeats — 3 per (arm, concurrency) cell, concurrency ∈ {1, 4, 8}. Median reported with spread, never a single run.
- Noise floor —
off_Aandoff_Bare the same configuration, booted first and last. Their difference is the floor, and because they are separate boots it includes boot-to-boot variance that three reps inside one boot cannot see. Runningoff_Blast also catches monotonic drift across the campaign. - Positive control (a) — MTP acceptance must be > 0 on every MTP arm, read
from
/metricsas a delta. A head that loads uninitialised serves fine and reports ~0% accept; that is the failure ourgenseat'sre:^mtp.*ignore-list footgun produces. An arm with ~0% acceptance is void — it measured a broken head, not MTP. (Pre-checked: this checkpoint excludesmtp.*andmodel.mtp.*from quantization in both quant configs, and its own audit reports all 31 MTP tensors byte-identical to source.) - Positive control (b) — aggregate throughput must rise with concurrency inside every arm. Known-true for a batching server; if the harness can't see it, the harness is blind and its negatives are worthless.
- Null control —
off_Avsoff_Bmust show no effect beyond the floor. - Stated sensitivity floor — the report prints "cannot resolve effects smaller
than X" from the observed
off_A/off_Bspread. A delta under it is not a finding.
Harness parity. Every arm's argv is derived from the live compose file via
docker compose config, not retyped, so the arms are provably identical except
for --speculative-config. That is what stops a stray flag from quietly becoming
the real independent variable.
Instruments
concbench.py— the house concurrency harness, reused unchanged fromservices/gen-seat-mixed-quant/bench/. It already avoids the three traps that have produced confident wrong answers here before: unseeded nonces so prefix caching can't fake prefill, MTP acceptance read as a delta so a long-lived seat's history doesn't swamp it, and aggregate throughput from wall clock rather than a sum of per-request rates.run-campaign.sh— the driver. Runs on fv-ml1 against127.0.0.1so the NH3↔FV mesh hop is not in the measurement.
Known gaps in this design
- No TTFT.
concbench.pydoesn't stream, so prefill latency isn't measured separately. The MTP mechanisms above are decode-side, so this is an acceptable omission — but it means the campaign cannot speak to first-token latency. - One prompt shape. ~400 output tokens from a short prompt. MTP acceptance is workload-dependent; a long-context or code workload could differ and this says nothing about them.
- The offload's contribution is not isolated. If MTP loses, this design cannot
separate mechanism 1 from mechanism 2. The clean discriminator is available and
cheap — re-run the best MTP arm at TP=2 across GPU 2+3 with
cpu_offload: falseso the table is resident, and compare the MTP delta with and without the PCIe path. GPU 3 is idle, so this is a follow-up worth doing if the answer matters.