docs(quant-playbook): acceptance is not throughput -- always run the depth control

Measured 2026-08-22 on one target with one instrument: raising MTP
num_speculative_tokens from 3 to 7 improved accepted length from 2.753 to
3.041 per forward pass while throughput fell from 114.9 to 74.0 tok/s.
Reporting acceptance alone would have recommended a 36% regression.

The cause is architectural rather than model-specific. A single-module MTP
head has no depth of its own, so vLLM runs it autoregressively and k draft
tokens cost k sequential forward passes. Past a shallow depth the drafting
cost exceeds what the extra accepted tokens save.

Records the comparison rule that follows: match k when comparing two
speculative methods, or the measurement is of depth rather than method. A
parallel-drafting drafter at k=7 against an autoregressive MTP at k=3 is not
a method comparison. In the case that produced this, the depth control
showed most of the apparent acceptance advantage was depth, while the
throughput advantage was real and came from parallel drafting -- our MTP was
better at position 0 and still lost overall.

Only the measured, model-agnostic result is recorded here. The
DFlash2-specific findings, the hypotheses that remain unproven, and the
wrong turns taken along the way live in
persistent-memory.d/2026-08-22-dflash2-spec-decode.md with explicit
epistemic labels, deliberately kept out of the playbook.
This commit is contained in:
2026-08-22 00:52:06 -07:00
parent 9f87b7c4e5
commit ab3a0ca5bc
+30
View File
@@ -541,6 +541,36 @@ thinking split, long-context needle, streaming), `serve_probe.sh`.
---
### 5.1 ⭐⭐ Acceptance is not throughput — always run the DEPTH control
**Measured 2026-08-22**, same instrument (vLLM's own `spec_decode` counters, delta over a fixed
workload, temp 0), same target, same engine:
| config | accepted tok/forward | throughput |
|---|---|---|
| MTP k=3 | 2.753 | 114.9 tok/s |
| MTP k=7 | **3.041** ⬆ | **74.0 tok/s** ⬇ |
**Raising `num_speculative_tokens` improved acceptance and destroyed throughput.** Reporting
acceptance alone would have recommended a 36% regression.
**Why:** a single-module MTP head (`mtp_num_hidden_layers: 1`, one `mtp.layers.0`) has no depth
of its own — vLLM runs it **autoregressively**, so k draft tokens cost **k sequential forward
passes**. Past a shallow depth the drafting cost exceeds what the extra accepted tokens save.
Check `mtp_num_hidden_layers` before assuming depth is cheap.
**The rule: when comparing two speculative methods, match k, or you are measuring depth rather
than method.** A parallel-drafting drafter (DFlash2 and kin, which propose a whole block in one
pass) at k=7 versus an autoregressive MTP at k=3 is not a method comparison — the depth control
is what separates them. In our case the control showed most of the apparent acceptance win was
depth, while the *throughput* win was real and came from parallel drafting, not better drafts:
our MTP was **better at position 0** (79.6% vs 75.4%) and still lost overall.
**Corollary — report both, always.** Acceptance rate, mean accepted length, and end-to-end
tok/s. Any one of the three alone can point the wrong way.
---
## 6. Hardware and co-residency
- **ana-ml2 = Blackwell (sm_120)**, 2× 96 GB. Native FP4 + FP8. Hopper-exact code paths