docs(quant-playbook): acceptance is not throughput -- always run the depth control
Measured 2026-08-22 on one target with one instrument: raising MTP num_speculative_tokens from 3 to 7 improved accepted length from 2.753 to 3.041 per forward pass while throughput fell from 114.9 to 74.0 tok/s. Reporting acceptance alone would have recommended a 36% regression. The cause is architectural rather than model-specific. A single-module MTP head has no depth of its own, so vLLM runs it autoregressively and k draft tokens cost k sequential forward passes. Past a shallow depth the drafting cost exceeds what the extra accepted tokens save. Records the comparison rule that follows: match k when comparing two speculative methods, or the measurement is of depth rather than method. A parallel-drafting drafter at k=7 against an autoregressive MTP at k=3 is not a method comparison. In the case that produced this, the depth control showed most of the apparent acceptance advantage was depth, while the throughput advantage was real and came from parallel drafting -- our MTP was better at position 0 and still lost overall. Only the measured, model-agnostic result is recorded here. The DFlash2-specific findings, the hypotheses that remain unproven, and the wrong turns taken along the way live in persistent-memory.d/2026-08-22-dflash2-spec-decode.md with explicit epistemic labels, deliberately kept out of the playbook.
This commit is contained in:
@@ -541,6 +541,36 @@ thinking split, long-context needle, streaming), `serve_probe.sh`.
|
||||
|
||||
---
|
||||
|
||||
### 5.1 ⭐⭐ Acceptance is not throughput — always run the DEPTH control
|
||||
|
||||
**Measured 2026-08-22**, same instrument (vLLM's own `spec_decode` counters, delta over a fixed
|
||||
workload, temp 0), same target, same engine:
|
||||
|
||||
| config | accepted tok/forward | throughput |
|
||||
|---|---|---|
|
||||
| MTP k=3 | 2.753 | 114.9 tok/s |
|
||||
| MTP k=7 | **3.041** ⬆ | **74.0 tok/s** ⬇ |
|
||||
|
||||
**Raising `num_speculative_tokens` improved acceptance and destroyed throughput.** Reporting
|
||||
acceptance alone would have recommended a 36% regression.
|
||||
|
||||
**Why:** a single-module MTP head (`mtp_num_hidden_layers: 1`, one `mtp.layers.0`) has no depth
|
||||
of its own — vLLM runs it **autoregressively**, so k draft tokens cost **k sequential forward
|
||||
passes**. Past a shallow depth the drafting cost exceeds what the extra accepted tokens save.
|
||||
Check `mtp_num_hidden_layers` before assuming depth is cheap.
|
||||
|
||||
**The rule: when comparing two speculative methods, match k, or you are measuring depth rather
|
||||
than method.** A parallel-drafting drafter (DFlash2 and kin, which propose a whole block in one
|
||||
pass) at k=7 versus an autoregressive MTP at k=3 is not a method comparison — the depth control
|
||||
is what separates them. In our case the control showed most of the apparent acceptance win was
|
||||
depth, while the *throughput* win was real and came from parallel drafting, not better drafts:
|
||||
our MTP was **better at position 0** (79.6% vs 75.4%) and still lost overall.
|
||||
|
||||
**Corollary — report both, always.** Acceptance rate, mean accepted length, and end-to-end
|
||||
tok/s. Any one of the three alone can point the wrong way.
|
||||
|
||||
---
|
||||
|
||||
## 6. Hardware and co-residency
|
||||
|
||||
- **ana-ml2 = Blackwell (sm_120)**, 2× 96 GB. Native FP4 + FP8. Hopper-exact code paths
|
||||
|
||||
Reference in New Issue
Block a user